== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:10:14 (1767319814) start mds service on oleg332-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 91 (time 1767319961.84 total 10.00 last 9.10) - open/close 225 (time 1767319972.51 total 20.67 last 12.56) - open/close 376 (time 1767319983.46 total 31.63 last 13.78) - open/close 501 (time 1767319994.30 total 42.46 last 11.54) - open/close 681 (time 1767320004.45 total 52.62 last 17.72) - open/close 911 (time 1767320014.47 total 62.63 last 22.97) - open/close 1168 (time 1767320027.17 total 75.33 last 20.24) - open/close 1390 (time 1767320037.17 total 85.33 last 22.19) - open/close 1599 (time 1767320047.98 total 96.15 last 19.32) - open/close 1757 (time 1767320058.02 total 106.18 last 15.74) - open/close 1936 (time 1767320068.46 total 116.62 last 17.15) - open/close 2070 (time 1767320078.63 total 126.79 last 13.17) - open/close 2296 (time 1767320088.64 total 136.80 last 22.59) - open/close 2530 (time 1767320099.10 total 147.26 last 22.36) - open/close 2734 (time 1767320109.88 total 158.04 last 18.93) - open/close 2953 (time 1767320119.88 total 168.05 last 21.89) - open/close 3119 (time 1767320130.52 total 178.69 last 15.60) - open/close 3289 (time 1767320140.91 total 189.07 last 16.37) - open/close 3472 (time 1767320150.91 total 199.08 last 18.29) - open/close 3589 (time 1767320160.93 total 209.09 last 11.68) - open/close 3723 (time 1767320170.94 total 219.11 last 13.38) - open/close 3971 (time 1767320181.13 total 229.30 last 24.33) - open/close 4119 (time 1767320191.14 total 239.31 last 14.79) - open/close 4256 (time 1767320201.17 total 249.34 last 13.66) - open/close 4322 (time 1767320211.20 total 259.37 last 6.58) - open/close 4535 (time 1767320221.36 total 269.53 last 20.97) - open/close 4738 (time 1767320232.02 total 280.19 last 19.04) - open/close 4772 (time 1767320243.09 total 291.25 last 3.07) - open/close 4853 (time 1767320253.22 total 301.39 last 7.99) - open/close 5051 (time 1767320263.84 total 312.01 last 18.65) - open/close 5185 (time 1767320273.85 total 322.01 last 13.40) - open/close 5270 (time 1767320284.44 total 332.61 last 8.02) - open/close 5397 (time 1767320294.61 total 342.77 last 12.49) - open/close 5557 (time 1767320304.63 total 352.80 last 15.96) - open/close 5622 (time 1767320314.64 total 362.81 last 6.49) - open/close 5792 (time 1767320324.65 total 372.82 last 16.98) - open/close 5962 (time 1767320335.12 total 383.28 last 16.24) - open/close 6122 (time 1767320345.12 total 393.29 last 16.00) - open/close 6263 (time 1767320355.28 total 403.44 last 13.88) - open/close 6398 (time 1767320365.62 total 413.78 last 13.06) - open/close 6508 (time 1767320376.34 total 424.51 last 10.26) - open/close 6764 (time 1767320386.35 total 434.52 last 25.57) - open/close 6956 (time 1767320396.90 total 445.07 last 18.20) - open/close 7166 (time 1767320406.91 total 455.07 last 20.99) - open/close 7339 (time 1767320416.92 total 465.08 last 17.28) - open/close 7426 (time 1767320428.32 total 476.48 last 7.63) - open/close 7504 (time 1767320438.87 total 487.03 last 7.39) - open/close 7684 (time 1767320448.92 total 497.08 last 17.91) - open/close 7880 (time 1767320461.74 total 509.90 last 15.29) - open/close 8080 (time 1767320472.14 total 520.30 last 19.22) - open/close 8266 (time 1767320482.24 total 530.40 last 18.42) - open/close 8485 (time 1767320492.62 total 540.79 last 21.09) - open/close 8597 (time 1767320502.83 total 550.99 last 10.97) - open/close 8801 (time 1767320512.85 total 561.01 last 20.36) - open/close 8958 (time 1767320523.80 total 571.96 last 14.34) - open/close 9138 (time 1767320533.81 total 581.98 last 17.97) - open/close 9249 (time 1767320544.21 total 592.37 last 10.68) - open/close 9446 (time 1767320554.87 total 603.04 last 18.48) - open/close 9598 (time 1767320565.46 total 613.63 last 14.35) - open/close 9765 (time 1767320575.47 total 623.63 last 16.69) - open/close 9907 (time 1767320587.48 total 635.65 last 11.82) - open/close 10000 (time 1767320594.99 total 643.15 last 12.39) - open/close 10179 (time 1767320605.10 total 653.26 last 17.70) - open/close 10442 (time 1767320615.90 total 664.06 last 24.35) - open/close 10612 (time 1767320626.65 total 674.81 last 15.81) - open/close 10774 (time 1767320637.66 total 685.82 last 14.71) - open/close 10920 (time 1767320647.67 total 695.83 last 14.59) - open/close 11073 (time 1767320658.52 total 706.69 last 14.10) - open/close 11303 (time 1767320668.56 total 716.72 last 22.92) - open/close 11536 (time 1767320678.58 total 726.74 last 23.25) - open/close 11713 (time 1767320689.16 total 737.32 last 16.73) - open/close 12019 (time 1767320700.23 total 748.39 last 27.65) - open/close 12329 (time 1767320710.24 total 758.40 last 30.96) - open/close 12532 (time 1767320720.80 total 768.96 last 19.23) - open/close 12767 (time 1767320731.77 total 779.93 last 21.42) - open/close 13002 (time 1767320742.04 total 790.20 last 22.88) - open/close 13149 (time 1767320752.07 total 800.24 last 14.65) - open/close 13424 (time 1767320762.10 total 810.27 last 27.42) - open/close 13703 (time 1767320775.07 total 823.23 last 21.52) - open/close 13916 (time 1767320785.63 total 833.79 last 20.16) - open/close 14150 (time 1767320795.65 total 843.81 last 23.36) - open/close 14360 (time 1767320806.19 total 854.36 last 19.91) - open/close 14610 (time 1767320816.28 total 864.44 last 24.79) - open/close 14842 (time 1767320826.29 total 874.46 last 23.17) - open/close 15037 (time 1767320836.30 total 884.46 last 19.49) - open/close 15210 (time 1767320846.68 total 894.84 last 16.66) - open/close 15393 (time 1767320856.68 total 904.85 last 18.29) - open/close 15612 (time 1767320867.55 total 915.71 last 20.16) - open/close 15793 (time 1767320877.79 total 925.95 last 17.68) - open/close 16024 (time 1767320887.80 total 935.96 last 23.07) - open/close 16240 (time 1767320897.80 total 945.97 last 21.59) - open/close 16490 (time 1767320908.18 total 956.34 last 24.10) - open/close 16722 (time 1767320918.19 total 966.35 last 23.17) - open/close 16930 (time 1767320928.27 total 976.44 last 20.62) - open/close 17162 (time 1767320938.29 total 986.45 last 23.17) - open/close 17414 (time 1767320949.37 total 997.54 last 22.74) - open/close 17678 (time 1767320959.38 total 1007.54 last 26.39) - open/close 17860 (time 1767320970.26 total 1018.42 last 16.73) - open/close 18064 (time 1767320980.30 total 1028.46 last 20.32) - open/close 18372 (time 1767320990.30 total 1038.47 last 30.78) - open/close 19023 (time 1767321000.31 total 1048.48 last 65.03) - open/close 19705 (time 1767321010.32 total 1058.49 last 68.13) - open/close 20000 (time 1767321014.69 total 1062.85 last 67.62) - open/close 20735 (time 1767321024.69 total 1072.85 last 73.49) - open/close 21516 (time 1767321034.70 total 1082.87 last 77.98) - open/close 22167 (time 1767321044.71 total 1092.87 last 65.08) - open/close 22912 (time 1767321054.71 total 1102.87 last 74.47) - open/close 23786 (time 1767321064.71 total 1112.88 last 87.39) - open/close 24490 (time 1767321074.72 total 1122.88 last 70.35) - open/close 25175 (time 1767321084.72 total 1132.89 last 68.46) - open/close 25807 (time 1767321094.74 total 1142.90 last 63.10) - open/close 26411 (time 1767321104.74 total 1152.91 last 60.40) - open/close 27121 (time 1767321114.74 total 1162.91 last 71.00) - open/close 27826 (time 1767321124.76 total 1172.92 last 70.39) - open/close 28590 (time 1767321134.76 total 1182.92 last 76.40) - open/close 29352 (time 1767321144.76 total 1192.92 last 76.19) - open/close 30000 (time 1767321153.82 total 1201.98 last 71.55) - open/close 30734 (time 1767321163.82 total 1211.98 last 73.38) - open/close 31399 (time 1767321173.87 total 1222.04 last 66.15) - open/close 32182 (time 1767321183.87 total 1232.04 last 78.29) - open/close 32912 (time 1767321193.88 total 1242.04 last 72.97) - open/close 33668 (time 1767321203.88 total 1252.05 last 75.56) - open/close 34325 (time 1767321213.90 total 1262.07 last 65.57) - open/close 35088 (time 1767321223.91 total 1272.07 last 76.24) - open/close 35819 (time 1767321233.93 total 1282.09 last 72.99) - open/close 36554 (time 1767321243.93 total 1292.09 last 73.47) - open/close 37156 (time 1767321253.94 total 1302.11 last 60.12) - open/close 37837 (time 1767321263.94 total 1312.11 last 68.09) - open/close 38700 (time 1767321273.95 total 1322.11 last 86.28) - open/close 39474 (time 1767321283.96 total 1332.12 last 77.33) - open/close 40000 (time 1767321291.79 total 1339.95 last 67.14) - open/close 40738 (time 1767321301.79 total 1349.96 last 73.78) - open/close 41529 (time 1767321311.81 total 1359.97 last 79.00) - open/close 42335 (time 1767321321.81 total 1369.98 last 80.53) - open/close 43008 (time 1767321331.82 total 1379.99 last 67.25) - open/close 43812 (time 1767321341.83 total 1389.99 last 80.35) - open/close 44545 (time 1767321351.84 total 1400.00 last 73.23) - open/close 45284 (time 1767321361.85 total 1410.01 last 73.83) - open/close 46012 (time 1767321371.86 total 1420.02 last 72.71) - open/close 46769 (time 1767321381.86 total 1430.03 last 75.68) - open/close 47621 (time 1767321391.88 total 1440.04 last 85.06) - open/close 48354 (time 1767321401.88 total 1450.05 last 73.27) - open/close 49086 (time 1767321411.89 total 1460.05 last 73.18) - open/close 49792 (time 1767321421.89 total 1470.05 last 70.59) total: 50000 open/close in 1473.16 seconds: 33.94 ops/second - unlinked 0 (time 1767321430 ; total 0 ; last 0) - unlinked 10000 (time 1767321568 ; total 138 ; last 138) - unlinked 20000 (time 1767321705 ; total 275 ; last 137) - unlinked 30000 (time 1767321844 ; total 414 ; last 139) - unlinked 40000 (time 1767321977 ; total 547 ; last 133) total: 50000 unlinks in 688 seconds: 72.674416 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2