== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:18:48 (1786735128) start mds service on oleg105-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 148 (time 1786735236.77 total 10.01 last 14.78) - open/close 269 (time 1786735247.62 total 20.86 last 11.15) - open/close 522 (time 1786735257.63 total 30.88 last 25.26) - open/close 678 (time 1786735267.65 total 40.89 last 15.57) - open/close 922 (time 1786735278.09 total 51.33 last 23.38) - open/close 1207 (time 1786735288.90 total 62.14 last 26.37) - open/close 1435 (time 1786735298.90 total 72.14 last 22.80) - open/close 1686 (time 1786735309.82 total 83.07 last 22.97) - open/close 1835 (time 1786735319.83 total 93.07 last 14.89) - open/close 2030 (time 1786735329.83 total 103.08 last 19.50) - open/close 2169 (time 1786735339.84 total 113.08 last 13.89) - open/close 2378 (time 1786735349.87 total 123.12 last 20.83) - open/close 2544 (time 1786735361.28 total 134.52 last 14.55) - open/close 2801 (time 1786735371.30 total 144.54 last 25.65) - open/close 2933 (time 1786735383.42 total 156.66 last 10.89) - open/close 3035 (time 1786735394.68 total 167.93 last 9.05) - open/close 3173 (time 1786735404.70 total 177.94 last 13.78) - open/close 3397 (time 1786735415.46 total 188.70 last 20.82) - open/close 3566 (time 1786735425.48 total 198.72 last 16.87) - open/close 3799 (time 1786735436.08 total 209.32 last 21.98) - open/close 4039 (time 1786735446.09 total 219.33 last 23.98) - open/close 4324 (time 1786735456.10 total 229.34 last 28.46) - open/close 4577 (time 1786735466.86 total 240.10 last 23.52) - open/close 4757 (time 1786735476.87 total 250.12 last 17.97) - open/close 5035 (time 1786735487.48 total 260.72 last 26.22) - open/close 5290 (time 1786735497.48 total 270.72 last 25.50) - open/close 5415 (time 1786735507.48 total 280.72 last 12.50) - open/close 5634 (time 1786735520.25 total 293.49 last 17.15) - open/close 5717 (time 1786735530.49 total 303.73 last 8.10) - open/close 5833 (time 1786735540.51 total 313.75 last 11.58) - open/close 5905 (time 1786735550.78 total 324.02 last 7.01) - open/close 6035 (time 1786735561.26 total 334.50 last 12.40) - open/close 6198 (time 1786735572.11 total 345.35 last 15.02) - open/close 6350 (time 1786735582.79 total 356.04 last 14.23) - open/close 6477 (time 1786735593.80 total 367.04 last 11.54) - open/close 6637 (time 1786735606.71 total 379.95 last 12.39) - open/close 6721 (time 1786735617.14 total 390.39 last 8.05) - open/close 6839 (time 1786735628.15 total 401.39 last 10.72) - open/close 6993 (time 1786735638.16 total 411.40 last 15.38) - open/close 7222 (time 1786735648.19 total 421.44 last 22.83) - open/close 7326 (time 1786735658.76 total 432.00 last 9.84) - open/close 7481 (time 1786735668.76 total 442.01 last 15.50) - open/close 7661 (time 1786735679.21 total 452.46 last 17.22) - open/close 7814 (time 1786735689.27 total 462.52 last 15.21) - open/close 7934 (time 1786735699.57 total 472.81 last 11.65) - open/close 8024 (time 1786735709.89 total 483.13 last 8.73) - open/close 8163 (time 1786735719.92 total 493.16 last 13.85) - open/close 8256 (time 1786735730.54 total 503.79 last 8.75) - open/close 8300 (time 1786735741.17 total 514.41 last 4.14) - open/close 8357 (time 1786735751.17 total 524.42 last 5.70) - open/close 8457 (time 1786735761.73 total 534.97 last 9.47) - open/close 8493 (time 1786735771.92 total 545.16 last 3.53) - open/close 8625 (time 1786735781.94 total 555.19 last 13.16) - open/close 8778 (time 1786735792.63 total 565.87 last 14.32) - open/close 8852 (time 1786735802.64 total 575.89 last 7.39) - open/close 9013 (time 1786735813.24 total 586.48 last 15.20) - open/close 9111 (time 1786735824.20 total 597.44 last 8.94) - open/close 9350 (time 1786735834.20 total 607.45 last 23.88) - open/close 9609 (time 1786735844.22 total 617.46 last 25.87) - open/close 9893 (time 1786735854.23 total 627.47 last 28.36) - open/close 10000 (time 1786735859.09 total 632.33 last 22.03) - open/close 10236 (time 1786735869.09 total 642.34 last 23.59) - open/close 10475 (time 1786735881.33 total 654.57 last 19.53) - open/close 10675 (time 1786735891.35 total 664.59 last 19.97) - open/close 10891 (time 1786735901.36 total 674.60 last 21.58) - open/close 11104 (time 1786735911.36 total 684.60 last 21.29) - open/close 11265 (time 1786735921.39 total 694.64 last 16.05) - open/close 11386 (time 1786735931.57 total 704.82 last 11.89) - open/close 11490 (time 1786735942.18 total 715.42 last 9.81) - open/close 11692 (time 1786735952.18 total 725.42 last 20.20) - open/close 11861 (time 1786735962.20 total 735.44 last 16.87) - open/close 11962 (time 1786735974.59 total 747.83 last 8.15) - open/close 12132 (time 1786735985.45 total 758.70 last 15.64) - open/close 12253 (time 1786735995.77 total 769.01 last 11.73) - open/close 12366 (time 1786736005.79 total 779.03 last 11.28) - open/close 12497 (time 1786736016.21 total 789.46 last 12.57) - open/close 12567 (time 1786736026.97 total 800.22 last 6.51) - open/close 12766 (time 1786736040.26 total 813.51 last 14.98) - open/close 12958 (time 1786736050.74 total 823.98 last 18.33) - open/close 13086 (time 1786736060.76 total 834.00 last 12.77) - open/close 13201 (time 1786736070.76 total 844.00 last 11.50) - open/close 13331 (time 1786736081.22 total 854.46 last 12.43) - open/close 13389 (time 1786736091.76 total 865.00 last 5.50) - open/close 13515 (time 1786736101.77 total 875.01 last 12.58) - open/close 13590 (time 1786736113.08 total 886.32 last 6.63) - open/close 13791 (time 1786736123.69 total 896.93 last 18.94) - open/close 13987 (time 1786736134.66 total 907.90 last 17.87) - open/close 14084 (time 1786736144.89 total 918.13 last 9.48) - open/close 14310 (time 1786736155.77 total 929.01 last 20.77) - open/close 14456 (time 1786736165.94 total 939.19 last 14.35) - open/close 14585 (time 1786736178.64 total 951.88 last 10.16) - open/close 14668 (time 1786736191.04 total 964.29 last 6.69) - open/close 14835 (time 1786736201.05 total 974.29 last 16.69) - open/close 14980 (time 1786736211.89 total 985.13 last 13.38) - open/close 15131 (time 1786736222.00 total 995.25 last 14.93) - open/close 15206 (time 1786736232.33 total 1005.57 last 7.27) - open/close 15328 (time 1786736242.38 total 1015.62 last 12.14) - open/close 15424 (time 1786736252.95 total 1026.20 last 9.08) - open/close 15537 (time 1786736265.92 total 1039.16 last 8.72) - open/close 15620 (time 1786736276.22 total 1049.46 last 8.06) - open/close 15760 (time 1786736286.79 total 1060.03 last 13.24) - open/close 15929 (time 1786736296.80 total 1070.04 last 16.89) - open/close 16672 (time 1786736306.83 total 1080.07 last 74.05) - open/close 17471 (time 1786736316.83 total 1090.07 last 79.88) - open/close 18157 (time 1786736326.85 total 1100.09 last 68.48) - open/close 18792 (time 1786736336.85 total 1110.09 last 63.48) - open/close 19511 (time 1786736346.85 total 1120.10 last 71.88) - open/close 20000 (time 1786736354.03 total 1127.27 last 68.19) - open/close 20796 (time 1786736364.03 total 1137.27 last 79.56) - open/close 21443 (time 1786736374.04 total 1147.28 last 64.64) - open/close 22132 (time 1786736384.05 total 1157.29 last 68.86) - open/close 22860 (time 1786736394.05 total 1167.29 last 72.75) - open/close 23490 (time 1786736404.06 total 1177.30 last 62.94) - open/close 23982 (time 1786736414.07 total 1187.31 last 49.18) - open/close 24484 (time 1786736424.08 total 1197.33 last 50.12) - open/close 24972 (time 1786736434.09 total 1207.33 last 48.78) - open/close 25671 (time 1786736444.10 total 1217.34 last 69.84) - open/close 26390 (time 1786736454.10 total 1227.34 last 71.90) - open/close 27116 (time 1786736464.10 total 1237.35 last 72.56) - open/close 27829 (time 1786736474.11 total 1247.35 last 71.27) - open/close 28576 (time 1786736484.12 total 1257.36 last 74.61) - open/close 29245 (time 1786736494.13 total 1267.37 last 66.85) - open/close 29903 (time 1786736504.15 total 1277.39 last 65.64) - open/close 30653 (time 1786736514.15 total 1287.40 last 74.98) - open/close 31395 (time 1786736524.16 total 1297.40 last 74.13) - open/close 32060 (time 1786736534.22 total 1307.46 last 66.14) - open/close 32564 (time 1786736544.26 total 1317.50 last 50.20) - open/close 33149 (time 1786736554.29 total 1327.53 last 58.33) - open/close 34073 (time 1786736564.30 total 1337.54 last 92.32) - open/close 34820 (time 1786736574.30 total 1347.54 last 74.65) - open/close 35388 (time 1786736584.31 total 1357.55 last 56.76) - open/close 35878 (time 1786736594.32 total 1367.56 last 48.97) - open/close 36390 (time 1786736604.35 total 1377.59 last 51.03) - open/close 37059 (time 1786736614.38 total 1387.62 last 66.73) - open/close 37665 (time 1786736624.38 total 1397.62 last 60.58) - open/close 38375 (time 1786736634.40 total 1407.64 last 70.87) - open/close 39112 (time 1786736644.41 total 1417.65 last 73.63) - open/close 39830 (time 1786736654.42 total 1427.67 last 71.67) - open/close 40000 (time 1786736656.81 total 1430.05 last 71.24) - open/close 40686 (time 1786736666.82 total 1440.06 last 68.53) - open/close 41494 (time 1786736676.83 total 1450.08 last 80.70) - open/close 42292 (time 1786736686.84 total 1460.08 last 79.78) - open/close 43028 (time 1786736696.84 total 1470.08 last 73.58) - open/close 43794 (time 1786736706.85 total 1480.09 last 76.53) - open/close 44612 (time 1786736716.86 total 1490.10 last 81.72) - open/close 45497 (time 1786736726.86 total 1500.10 last 88.48) - open/close 46230 (time 1786736736.86 total 1510.10 last 73.29) - open/close 46991 (time 1786736746.87 total 1520.11 last 76.06) - open/close 47820 (time 1786736756.87 total 1530.11 last 82.86) - open/close 48609 (time 1786736766.89 total 1540.14 last 78.72) - open/close 49378 (time 1786736776.90 total 1550.14 last 76.84) total: 50000 open/close in 1558.77 seconds: 32.08 ops/second - unlinked 0 (time 1786736790 ; total 0 ; last 0) - unlinked 10000 (time 1786736937 ; total 147 ; last 147) - unlinked 20000 (time 1786737101 ; total 311 ; last 164) - unlinked 30000 (time 1786737260 ; total 470 ; last 159) - unlinked 40000 (time 1786737438 ; total 648 ; last 178) total: 50000 unlinks in 804 seconds: 62.189056 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2