== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:45:41 (1786815941) start mds service on oleg237-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 109 (time 1786816034.83 total 10.00 last 10.90) - open/close 331 (time 1786816045.31 total 20.48 last 21.18) - open/close 453 (time 1786816055.37 total 30.53 last 12.14) - open/close 595 (time 1786816066.73 total 41.90 last 12.49) - open/close 756 (time 1786816076.74 total 51.91 last 16.09) - open/close 904 (time 1786816087.51 total 62.68 last 13.73) - open/close 994 (time 1786816097.80 total 72.97 last 8.75) - open/close 1178 (time 1786816108.64 total 83.81 last 16.96) - open/close 1320 (time 1786816119.07 total 94.24 last 13.62) - open/close 1488 (time 1786816129.80 total 104.97 last 15.65) - open/close 1663 (time 1786816139.83 total 115.00 last 17.46) - open/close 1770 (time 1786816150.65 total 125.82 last 9.89) - open/close 1889 (time 1786816160.74 total 135.91 last 11.79) - open/close 2058 (time 1786816170.75 total 145.92 last 16.88) - open/close 2162 (time 1786816180.77 total 155.94 last 10.38) - open/close 2243 (time 1786816191.76 total 166.93 last 7.37) - open/close 2371 (time 1786816202.50 total 177.67 last 11.92) - open/close 2593 (time 1786816213.32 total 188.49 last 20.51) - open/close 2801 (time 1786816223.38 total 198.55 last 20.67) - open/close 2924 (time 1786816233.98 total 209.15 last 11.60) - open/close 3018 (time 1786816243.99 total 219.16 last 9.40) - open/close 3143 (time 1786816255.38 total 230.55 last 10.97) - open/close 3254 (time 1786816265.74 total 240.91 last 10.71) - open/close 3423 (time 1786816276.66 total 251.83 last 15.47) - open/close 3494 (time 1786816286.86 total 262.03 last 6.96) - open/close 3610 (time 1786816297.80 total 272.97 last 10.60) - open/close 3654 (time 1786816308.36 total 283.52 last 4.17) - open/close 3815 (time 1786816318.42 total 293.59 last 16.00) - open/close 4048 (time 1786816328.44 total 303.61 last 23.26) - open/close 4278 (time 1786816339.26 total 314.42 last 21.26) - open/close 4413 (time 1786816349.37 total 324.54 last 13.35) - open/close 4607 (time 1786816359.85 total 335.02 last 18.50) - open/close 4836 (time 1786816370.62 total 345.79 last 21.27) - open/close 5002 (time 1786816380.62 total 355.79 last 16.59) - open/close 5191 (time 1786816390.78 total 365.95 last 18.60) - open/close 5376 (time 1786816400.79 total 375.95 last 18.49) - open/close 5429 (time 1786816411.27 total 386.44 last 5.05) - open/close 5615 (time 1786816421.29 total 396.46 last 18.57) - open/close 5747 (time 1786816431.86 total 407.03 last 12.49) - open/close 5953 (time 1786816441.87 total 417.04 last 20.58) - open/close 6150 (time 1786816452.78 total 427.95 last 18.05) - open/close 6333 (time 1786816462.80 total 437.97 last 18.26) - open/close 6484 (time 1786816473.66 total 448.83 last 13.91) - open/close 6587 (time 1786816484.08 total 459.25 last 9.88) - open/close 6771 (time 1786816494.79 total 469.96 last 17.19) - open/close 6964 (time 1786816505.24 total 480.41 last 18.47) - open/close 7186 (time 1786816516.08 total 491.25 last 20.47) - open/close 7330 (time 1786816526.15 total 501.32 last 14.30) - open/close 7482 (time 1786816536.38 total 511.55 last 14.86) - open/close 7542 (time 1786816546.39 total 521.56 last 5.99) - open/close 7652 (time 1786816556.61 total 531.78 last 10.76) - open/close 7812 (time 1786816566.63 total 541.80 last 15.98) - open/close 7979 (time 1786816577.04 total 552.21 last 16.03) - open/close 8098 (time 1786816588.05 total 563.21 last 10.81) - open/close 8209 (time 1786816598.53 total 573.70 last 10.59) - open/close 8307 (time 1786816608.53 total 583.70 last 9.80) - open/close 8434 (time 1786816619.19 total 594.36 last 11.91) - open/close 8528 (time 1786816631.88 total 607.05 last 7.41) - open/close 8640 (time 1786816641.89 total 617.06 last 11.19) - open/close 8736 (time 1786816651.94 total 627.10 last 9.55) - open/close 8909 (time 1786816662.47 total 637.63 last 16.43) - open/close 9070 (time 1786816672.78 total 647.95 last 15.61) - open/close 9215 (time 1786816683.33 total 658.50 last 13.74) - open/close 9328 (time 1786816693.43 total 668.60 last 11.19) - open/close 9461 (time 1786816703.99 total 679.16 last 12.60) - open/close 9574 (time 1786816714.96 total 690.13 last 10.31) - open/close 9728 (time 1786816724.96 total 700.13 last 15.39) - open/close 9847 (time 1786816734.98 total 710.15 last 11.87) - open/close 9978 (time 1786816745.85 total 721.02 last 12.06) - open/close 10000 (time 1786816749.57 total 724.74 last 5.91) - open/close 10195 (time 1786816760.41 total 735.58 last 18.00) - open/close 10342 (time 1786816771.06 total 746.23 last 13.80) - open/close 10442 (time 1786816781.71 total 756.88 last 9.39) - open/close 10574 (time 1786816792.44 total 767.61 last 12.30) - open/close 10697 (time 1786816803.10 total 778.27 last 11.54) - open/close 10798 (time 1786816813.12 total 788.29 last 10.08) - open/close 10893 (time 1786816823.93 total 799.10 last 8.78) - open/close 11002 (time 1786816833.97 total 809.14 last 10.86) - open/close 11115 (time 1786816843.98 total 819.15 last 11.29) - open/close 11254 (time 1786816854.25 total 829.42 last 13.54) - open/close 11359 (time 1786816865.00 total 840.17 last 9.76) - open/close 11492 (time 1786816875.01 total 850.18 last 13.30) - open/close 11567 (time 1786816885.64 total 860.81 last 7.05) - open/close 11670 (time 1786816895.65 total 870.82 last 10.29) - open/close 11749 (time 1786816905.65 total 880.82 last 7.90) - open/close 11856 (time 1786816915.76 total 890.93 last 10.59) - open/close 12018 (time 1786816926.47 total 901.64 last 15.13) - open/close 12134 (time 1786816936.47 total 911.64 last 11.59) - open/close 12200 (time 1786816946.95 total 922.12 last 6.30) - open/close 12254 (time 1786816956.95 total 932.12 last 5.40) - open/close 12319 (time 1786816967.21 total 942.38 last 6.34) - open/close 12427 (time 1786816977.67 total 952.83 last 10.33) - open/close 12597 (time 1786816988.02 total 963.19 last 16.42) - open/close 12773 (time 1786816998.03 total 973.20 last 17.58) - open/close 12875 (time 1786817008.95 total 984.12 last 9.34) - open/close 13079 (time 1786817019.51 total 994.68 last 19.32) - open/close 13290 (time 1786817030.47 total 1005.64 last 19.25) - open/close 13388 (time 1786817041.20 total 1016.37 last 9.13) - open/close 13548 (time 1786817052.16 total 1027.33 last 14.60) - open/close 13662 (time 1786817062.21 total 1037.38 last 11.34) - open/close 13774 (time 1786817072.50 total 1047.67 last 10.88) - open/close 13877 (time 1786817082.52 total 1057.69 last 10.29) - open/close 14019 (time 1786817092.58 total 1067.75 last 14.12) - open/close 14101 (time 1786817103.87 total 1079.04 last 7.26) - open/close 14186 (time 1786817115.25 total 1090.42 last 7.47) - open/close 14293 (time 1786817125.26 total 1100.43 last 10.69) - open/close 14419 (time 1786817135.27 total 1110.44 last 12.58) - open/close 14481 (time 1786817145.29 total 1120.46 last 6.19) - open/close 14599 (time 1786817155.55 total 1130.72 last 11.50) - open/close 14762 (time 1786817165.56 total 1140.73 last 16.28) - open/close 14892 (time 1786817175.60 total 1150.77 last 12.95) - open/close 15248 (time 1786817185.60 total 1160.77 last 35.57) - open/close 15923 (time 1786817195.62 total 1170.79 last 67.40) - open/close 16668 (time 1786817205.62 total 1180.79 last 74.49) - open/close 17344 (time 1786817215.63 total 1190.80 last 67.56) - open/close 17816 (time 1786817225.67 total 1200.84 last 47.01) - open/close 18443 (time 1786817235.69 total 1210.86 last 62.58) - open/close 19092 (time 1786817245.70 total 1220.87 last 64.84) - open/close 19633 (time 1786817255.70 total 1230.87 last 54.09) - open/close 20000 (time 1786817262.24 total 1237.41 last 56.06) - open/close 20567 (time 1786817272.25 total 1247.42 last 56.68) - open/close 21206 (time 1786817282.26 total 1257.43 last 63.85) - open/close 21854 (time 1786817292.26 total 1267.43 last 64.76) - open/close 22542 (time 1786817302.27 total 1277.44 last 68.77) - open/close 23188 (time 1786817312.27 total 1287.44 last 64.58) - open/close 23808 (time 1786817322.27 total 1297.44 last 61.99) - open/close 24427 (time 1786817332.27 total 1307.44 last 61.90) - open/close 25023 (time 1786817342.29 total 1317.46 last 59.50) - open/close 25746 (time 1786817352.29 total 1327.46 last 72.26) - open/close 26499 (time 1786817362.30 total 1337.47 last 75.28) - open/close 27185 (time 1786817372.30 total 1347.47 last 68.56) - open/close 27852 (time 1786817382.31 total 1357.48 last 66.67) - open/close 28518 (time 1786817392.32 total 1367.49 last 66.52) - open/close 29156 (time 1786817402.33 total 1377.50 last 63.76) - open/close 29785 (time 1786817412.33 total 1387.50 last 62.87) - open/close 30000 (time 1786817414.97 total 1390.14 last 81.43) - open/close 30676 (time 1786817425.00 total 1400.17 last 67.39) - open/close 31325 (time 1786817435.03 total 1410.20 last 64.73) - open/close 31971 (time 1786817445.04 total 1420.21 last 64.54) - open/close 32554 (time 1786817455.04 total 1430.21 last 58.27) - open/close 33224 (time 1786817465.06 total 1440.22 last 66.92) - open/close 33908 (time 1786817475.06 total 1450.23 last 68.34) - open/close 34500 (time 1786817485.08 total 1460.25 last 59.13) - open/close 35090 (time 1786817495.09 total 1470.26 last 58.92) - open/close 35620 (time 1786817505.12 total 1480.29 last 52.86) - open/close 36222 (time 1786817515.13 total 1490.30 last 60.09) - open/close 36808 (time 1786817525.14 total 1500.31 last 58.54) - open/close 37511 (time 1786817535.16 total 1510.33 last 70.15) - open/close 38349 (time 1786817545.17 total 1520.34 last 83.79) - open/close 39194 (time 1786817555.17 total 1530.34 last 84.49) - open/close 39985 (time 1786817565.21 total 1540.37 last 78.80) - open/close 40594 (time 1786817575.21 total 1550.38 last 60.87) - open/close 41266 (time 1786817585.23 total 1560.40 last 67.05) - open/close 42159 (time 1786817595.23 total 1570.40 last 89.29) - open/close 42813 (time 1786817605.23 total 1580.40 last 65.40) - open/close 43388 (time 1786817615.24 total 1590.41 last 57.47) - open/close 44002 (time 1786817625.25 total 1600.42 last 61.35) - open/close 44686 (time 1786817635.26 total 1610.43 last 68.33) - open/close 45334 (time 1786817645.30 total 1620.47 last 64.55) - open/close 46039 (time 1786817655.31 total 1630.48 last 70.40) - open/close 46783 (time 1786817665.34 total 1640.50 last 74.22) - open/close 47384 (time 1786817675.34 total 1650.51 last 60.09) - open/close 48149 (time 1786817685.34 total 1660.51 last 76.45) - open/close 48980 (time 1786817695.35 total 1670.52 last 83.03) - open/close 49869 (time 1786817705.36 total 1680.53 last 88.86) total: 50000 open/close in 1682.29 seconds: 29.72 ops/second - unlinked 0 (time 1786817711 ; total 0 ; last 0) - unlinked 10000 (time 1786817839 ; total 128 ; last 128) - unlinked 20000 (time 1786817974 ; total 263 ; last 135) - unlinked 30000 (time 1786818108 ; total 397 ; last 134) - unlinked 40000 (time 1786818246 ; total 535 ; last 138) total: 50000 unlinks in 666 seconds: 75.075073 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2