== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:09:30 (1785391770) start mds service on oleg311-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1785391887.95 total 10.86 last 14.27) - open/close 273 (time 1785391898.95 total 21.86 last 10.73) - open/close 416 (time 1785391908.96 total 31.87 last 14.28) - open/close 509 (time 1785391918.98 total 41.90 last 9.28) - open/close 592 (time 1785391929.00 total 51.91 last 8.29) - open/close 651 (time 1785391939.01 total 61.92 last 5.89) - open/close 749 (time 1785391949.88 total 72.79 last 9.02) - open/close 876 (time 1785391960.64 total 83.56 last 11.80) - open/close 973 (time 1785391970.65 total 93.56 last 9.69) - open/close 1069 (time 1785391980.72 total 103.63 last 9.53) - open/close 1153 (time 1785391990.73 total 113.64 last 8.39) - open/close 1264 (time 1785392000.75 total 123.66 last 11.08) - open/close 1377 (time 1785392011.07 total 133.98 last 10.95) - open/close 1527 (time 1785392021.09 total 144.01 last 14.96) - open/close 1630 (time 1785392031.89 total 154.80 last 9.54) - open/close 1772 (time 1785392041.90 total 164.81 last 14.19) - open/close 1838 (time 1785392052.86 total 175.77 last 6.02) - open/close 2011 (time 1785392065.00 total 187.91 last 14.25) - open/close 2113 (time 1785392075.02 total 197.93 last 10.18) - open/close 2203 (time 1785392085.55 total 208.46 last 8.55) - open/close 2311 (time 1785392095.55 total 218.47 last 10.79) - open/close 2438 (time 1785392105.85 total 228.76 last 12.34) - open/close 2608 (time 1785392118.33 total 241.25 last 13.61) - open/close 2693 (time 1785392128.63 total 251.54 last 8.25) - open/close 2753 (time 1785392138.64 total 261.55 last 5.99) - open/close 2908 (time 1785392148.66 total 271.57 last 15.48) - open/close 3033 (time 1785392159.38 total 282.29 last 11.66) - open/close 3116 (time 1785392169.66 total 292.57 last 8.08) - open/close 3248 (time 1785392180.23 total 303.14 last 12.48) - open/close 3338 (time 1785392191.76 total 314.67 last 7.81) - open/close 3471 (time 1785392201.80 total 324.71 last 13.24) - open/close 3575 (time 1785392211.84 total 334.75 last 10.36) - open/close 3707 (time 1785392222.65 total 345.56 last 12.22) - open/close 3841 (time 1785392233.60 total 356.52 last 12.23) - open/close 3926 (time 1785392243.90 total 366.81 last 8.26) - open/close 4034 (time 1785392253.91 total 376.82 last 10.79) - open/close 4130 (time 1785392264.14 total 387.05 last 9.38) - open/close 4317 (time 1785392274.60 total 397.51 last 17.88) - open/close 4457 (time 1785392285.68 total 408.59 last 12.63) - open/close 4578 (time 1785392296.11 total 419.03 last 11.60) - open/close 4657 (time 1785392306.94 total 429.85 last 7.30) - open/close 4748 (time 1785392317.31 total 440.22 last 8.77) - open/close 4813 (time 1785392327.68 total 450.60 last 6.27) - open/close 4878 (time 1785392338.55 total 461.46 last 5.98) - open/close 4938 (time 1785392349.48 total 472.40 last 5.49) - open/close 4971 (time 1785392361.65 total 484.56 last 2.71) - open/close 5114 (time 1785392372.36 total 495.28 last 13.35) - open/close 5205 (time 1785392382.89 total 505.80 last 8.65) - open/close 5255 (time 1785392393.57 total 516.48 last 4.68) - open/close 5366 (time 1785392403.82 total 526.73 last 10.83) - open/close 5475 (time 1785392413.85 total 536.77 last 10.86) - open/close 5583 (time 1785392424.64 total 547.55 last 10.01) - open/close 5684 (time 1785392434.65 total 557.56 last 10.09) - open/close 5844 (time 1785392444.85 total 567.76 last 15.69) - open/close 5891 (time 1785392457.53 total 580.44 last 3.71) - open/close 6033 (time 1785392468.36 total 591.28 last 13.10) - open/close 6113 (time 1785392478.37 total 601.28 last 8.00) - open/close 6234 (time 1785392488.38 total 611.29 last 12.09) - open/close 6354 (time 1785392498.75 total 621.67 last 11.57) - open/close 6422 (time 1785392510.15 total 633.06 last 5.97) - open/close 6487 (time 1785392520.17 total 643.08 last 6.49) - open/close 6614 (time 1785392530.93 total 653.85 last 11.80) - open/close 6713 (time 1785392541.17 total 664.08 last 9.67) - open/close 6825 (time 1785392551.18 total 674.09 last 11.19) - open/close 6937 (time 1785392562.21 total 685.12 last 10.15) - open/close 7095 (time 1785392573.21 total 696.13 last 14.36) - open/close 7186 (time 1785392583.57 total 706.48 last 8.79) - open/close 7266 (time 1785392593.58 total 716.49 last 7.99) - open/close 7454 (time 1785392604.09 total 727.00 last 17.89) - open/close 7596 (time 1785392615.05 total 737.96 last 12.95) - open/close 7704 (time 1785392625.52 total 748.43 last 10.31) - open/close 7782 (time 1785392636.16 total 759.07 last 7.34) - open/close 7851 (time 1785392646.38 total 769.29 last 6.75) - open/close 7997 (time 1785392657.35 total 780.26 last 13.32) - open/close 8097 (time 1785392668.09 total 791.00 last 9.31) - open/close 8197 (time 1785392678.74 total 801.65 last 9.39) - open/close 8355 (time 1785392689.19 total 812.11 last 15.11) - open/close 8499 (time 1785392699.25 total 822.16 last 14.32) - open/close 8625 (time 1785392709.73 total 832.64 last 12.02) - open/close 8744 (time 1785392721.72 total 844.64 last 9.92) - open/close 8927 (time 1785392731.73 total 854.65 last 18.28) - open/close 9036 (time 1785392741.75 total 864.66 last 10.89) - open/close 9152 (time 1785392751.78 total 874.70 last 11.56) - open/close 9327 (time 1785392762.54 total 885.46 last 16.26) - open/close 9469 (time 1785392773.13 total 896.05 last 13.41) - open/close 9563 (time 1785392783.46 total 906.37 last 9.10) - open/close 9652 (time 1785392793.49 total 916.40 last 8.88) - open/close 9752 (time 1785392803.77 total 926.68 last 9.72) - open/close 9871 (time 1785392814.26 total 937.17 last 11.35) - open/close 10000 (time 1785392822.79 total 945.70 last 15.13) - open/close 10066 (time 1785392833.19 total 956.11 last 6.34) - open/close 10160 (time 1785392844.29 total 967.20 last 8.47) - open/close 10289 (time 1785392854.30 total 977.21 last 12.89) - open/close 10380 (time 1785392864.94 total 987.86 last 8.55) - open/close 10469 (time 1785392874.96 total 997.87 last 8.89) - open/close 10602 (time 1785392884.97 total 1007.89 last 13.28) - open/close 10737 (time 1785392895.00 total 1017.92 last 13.46) - open/close 10827 (time 1785392905.02 total 1027.93 last 8.99) - open/close 10938 (time 1785392915.62 total 1038.53 last 10.47) - open/close 11043 (time 1785392925.69 total 1048.60 last 10.43) - open/close 11136 (time 1785392935.74 total 1058.65 last 9.25) - open/close 11277 (time 1785392946.04 total 1068.95 last 13.69) - open/close 11419 (time 1785392956.06 total 1078.98 last 14.17) - open/close 11559 (time 1785392968.11 total 1091.03 last 11.62) - open/close 11644 (time 1785392980.52 total 1103.44 last 6.85) - open/close 11677 (time 1785392990.77 total 1113.68 last 3.22) - open/close 11739 (time 1785393000.79 total 1123.70 last 6.19) - open/close 11800 (time 1785393010.86 total 1133.77 last 6.06) - open/close 11907 (time 1785393020.86 total 1143.77 last 10.70) - open/close 11991 (time 1785393031.03 total 1153.94 last 8.26) - open/close 12067 (time 1785393045.24 total 1168.15 last 5.35) - open/close 12136 (time 1785393055.60 total 1178.52 last 6.66) - open/close 12210 (time 1785393065.92 total 1188.83 last 7.17) - open/close 12367 (time 1785393075.95 total 1198.86 last 15.65) - open/close 12499 (time 1785393086.57 total 1209.48 last 12.43) - open/close 12535 (time 1785393096.90 total 1219.81 last 3.48) - open/close 12643 (time 1785393107.87 total 1230.79 last 9.84) - open/close 12810 (time 1785393117.99 total 1240.90 last 16.51) - open/close 12909 (time 1785393129.87 total 1252.78 last 8.34) - open/close 13023 (time 1785393140.28 total 1263.20 last 10.94) - open/close 13171 (time 1785393150.72 total 1273.64 last 14.17) - open/close 13273 (time 1785393162.23 total 1285.15 last 8.86) - open/close 13672 (time 1785393172.26 total 1295.17 last 39.80) - open/close 14088 (time 1785393182.28 total 1305.19 last 41.51) - open/close 14524 (time 1785393192.28 total 1315.19 last 43.60) - open/close 15074 (time 1785393202.28 total 1325.19 last 54.99) - open/close 15560 (time 1785393212.28 total 1335.20 last 48.59) - open/close 16165 (time 1785393222.30 total 1345.21 last 60.41) - open/close 16789 (time 1785393232.31 total 1355.22 last 62.33) - open/close 17203 (time 1785393242.32 total 1365.23 last 41.37) - open/close 17682 (time 1785393252.32 total 1375.24 last 47.87) - open/close 18210 (time 1785393262.33 total 1385.24 last 52.79) - open/close 18811 (time 1785393272.34 total 1395.25 last 60.02) - open/close 19340 (time 1785393282.36 total 1405.27 last 52.80) - open/close 20000 (time 1785393291.44 total 1414.35 last 72.68) - open/close 20651 (time 1785393301.44 total 1424.35 last 65.10) - open/close 21108 (time 1785393311.45 total 1434.36 last 45.68) - open/close 21551 (time 1785393321.47 total 1444.38 last 44.18) - open/close 22063 (time 1785393331.48 total 1454.39 last 51.18) - open/close 22606 (time 1785393341.48 total 1464.40 last 54.26) - open/close 23135 (time 1785393351.49 total 1474.40 last 52.87) - open/close 23671 (time 1785393361.51 total 1484.42 last 53.47) - open/close 24308 (time 1785393371.51 total 1494.43 last 63.68) - open/close 24821 (time 1785393381.52 total 1504.43 last 51.27) - open/close 25277 (time 1785393391.55 total 1514.47 last 45.44) - open/close 25760 (time 1785393401.57 total 1524.49 last 48.21) - open/close 26253 (time 1785393411.59 total 1534.50 last 49.24) - open/close 26759 (time 1785393421.59 total 1544.51 last 50.56) - open/close 27431 (time 1785393431.60 total 1554.51 last 67.18) - open/close 27719 (time 1785393441.60 total 1564.52 last 28.78) - open/close 28334 (time 1785393451.61 total 1574.52 last 61.47) - open/close 28743 (time 1785393461.66 total 1584.58 last 40.68) - open/close 29086 (time 1785393471.67 total 1594.59 last 34.27) - open/close 29385 (time 1785393481.69 total 1604.60 last 29.85) - open/close 29717 (time 1785393491.76 total 1614.67 last 32.97) - open/close 30000 (time 1785393498.44 total 1621.35 last 42.37) - open/close 30553 (time 1785393508.45 total 1631.36 last 55.22) - open/close 31163 (time 1785393518.46 total 1641.37 last 60.97) - open/close 31659 (time 1785393528.46 total 1651.38 last 49.56) - open/close 32157 (time 1785393538.47 total 1661.38 last 49.77) - open/close 32682 (time 1785393548.48 total 1671.39 last 52.45) - open/close 33359 (time 1785393558.48 total 1681.39 last 67.70) - open/close 33989 (time 1785393568.49 total 1691.41 last 62.90) - open/close 34525 (time 1785393578.54 total 1701.45 last 53.36) - open/close 35000 (time 1785393588.55 total 1711.46 last 47.45) - open/close 35511 (time 1785393598.58 total 1721.49 last 50.96) - open/close 35966 (time 1785393608.58 total 1731.50 last 45.47) - open/close 36392 (time 1785393618.63 total 1741.54 last 42.42) - open/close 36935 (time 1785393628.63 total 1751.55 last 54.26) - open/close 37352 (time 1785393638.64 total 1761.55 last 41.67) - open/close 37751 (time 1785393648.68 total 1771.59 last 39.75) - open/close 38216 (time 1785393658.69 total 1781.60 last 46.44) - open/close 38696 (time 1785393668.69 total 1791.61 last 47.99) - open/close 39247 (time 1785393678.70 total 1801.62 last 55.05) - open/close 39732 (time 1785393688.71 total 1811.62 last 48.49) - open/close 40000 (time 1785393692.81 total 1815.72 last 65.30) - open/close 40462 (time 1785393702.83 total 1825.74 last 46.12) - open/close 40945 (time 1785393712.84 total 1835.76 last 48.21) - open/close 41471 (time 1785393722.85 total 1845.76 last 52.57) - open/close 41930 (time 1785393732.86 total 1855.77 last 45.86) - open/close 42537 (time 1785393742.87 total 1865.79 last 60.61) - open/close 43090 (time 1785393752.89 total 1875.81 last 55.19) - open/close 43740 (time 1785393762.91 total 1885.82 last 64.89) - open/close 44543 (time 1785393772.92 total 1895.83 last 80.24) - open/close 45166 (time 1785393782.94 total 1905.85 last 62.17) - open/close 45654 (time 1785393792.95 total 1915.87 last 48.73) - open/close 46045 (time 1785393802.98 total 1925.89 last 39.02) - open/close 46454 (time 1785393812.99 total 1935.90 last 40.86) - open/close 47007 (time 1785393822.99 total 1945.91 last 55.26) - open/close 47397 (time 1785393833.02 total 1955.93 last 38.91) - open/close 47758 (time 1785393843.03 total 1965.94 last 36.06) - open/close 48193 (time 1785393853.04 total 1975.95 last 43.44) - open/close 48688 (time 1785393863.06 total 1985.97 last 49.43) - open/close 49140 (time 1785393873.07 total 1995.99 last 45.11) - open/close 49526 (time 1785393883.10 total 2006.01 last 38.50) - open/close 49963 (time 1785393893.11 total 2016.02 last 43.67) total: 50000 open/close in 2016.32 seconds: 24.80 ops/second - unlinked 0 (time 1785393899 ; total 0 ; last 0) - unlinked 10000 (time 1785394114 ; total 215 ; last 215) - unlinked 20000 (time 1785394319 ; total 420 ; last 205) - unlinked 30000 (time 1785394545 ; total 646 ; last 226) - unlinked 40000 (time 1785394760 ; total 861 ; last 215) total: 50000 unlinks in 1058 seconds: 47.258980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2