== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:48:12 (1761238092) start mds service on oleg254-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg254-server: detected 4 online CPUs by sysfs oleg254-server: MODOPTS_LIBCFS= oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 119 (time 1761238164.57 total 10.01 last 11.89) - open/close 232 (time 1761238174.60 total 20.03 last 11.27) - open/close 330 (time 1761238184.60 total 30.03 last 9.80) - open/close 404 (time 1761238195.29 total 40.72 last 6.93) - open/close 563 (time 1761238206.23 total 51.66 last 14.53) - open/close 765 (time 1761238216.61 total 62.05 last 19.44) - open/close 919 (time 1761238227.35 total 72.78 last 14.35) - open/close 1000 (time 1761238237.60 total 83.03 last 7.90) - open/close 1123 (time 1761238247.73 total 93.16 last 12.15) - open/close 1344 (time 1761238259.59 total 105.02 last 18.64) - open/close 1558 (time 1761238269.96 total 115.39 last 20.63) - open/close 1766 (time 1761238280.58 total 126.02 last 19.58) - open/close 1967 (time 1761238290.64 total 136.08 last 19.98) - open/close 2137 (time 1761238300.65 total 146.08 last 16.98) - open/close 2364 (time 1761238310.67 total 156.11 last 22.65) - open/close 2574 (time 1761238320.84 total 166.27 last 20.66) - open/close 2759 (time 1761238331.54 total 176.97 last 17.28) - open/close 2963 (time 1761238342.00 total 187.44 last 19.49) - open/close 3184 (time 1761238353.06 total 198.50 last 19.98) - open/close 3375 (time 1761238363.93 total 209.37 last 17.58) - open/close 3469 (time 1761238374.68 total 220.11 last 8.75) - open/close 3600 (time 1761238384.70 total 230.13 last 13.07) - open/close 3732 (time 1761238395.15 total 240.58 last 12.63) - open/close 3898 (time 1761238405.97 total 251.40 last 15.34) - open/close 4015 (time 1761238415.98 total 261.42 last 11.68) - open/close 4249 (time 1761238426.76 total 272.20 last 21.71) - open/close 4497 (time 1761238436.78 total 282.21 last 24.76) - open/close 4721 (time 1761238447.51 total 292.95 last 20.86) - open/close 4961 (time 1761238458.43 total 303.87 last 21.98) - open/close 5064 (time 1761238469.11 total 314.55 last 9.65) - open/close 5280 (time 1761238479.17 total 324.61 last 21.47) - open/close 5506 (time 1761238489.18 total 334.61 last 22.58) - open/close 5655 (time 1761238499.23 total 344.66 last 14.83) - open/close 5869 (time 1761238509.24 total 354.67 last 21.37) - open/close 5988 (time 1761238519.36 total 364.79 last 11.76) - open/close 6103 (time 1761238529.38 total 374.81 last 11.48) - open/close 6262 (time 1761238539.39 total 384.82 last 15.88) - open/close 6346 (time 1761238549.70 total 395.14 last 8.14) - open/close 6500 (time 1761238560.46 total 405.90 last 14.31) - open/close 6645 (time 1761238570.83 total 416.26 last 13.99) - open/close 6866 (time 1761238580.83 total 426.27 last 22.09) - open/close 6964 (time 1761238591.43 total 436.86 last 9.25) - open/close 7101 (time 1761238601.75 total 447.19 last 13.27) - open/close 7312 (time 1761238612.53 total 457.96 last 19.59) - open/close 7544 (time 1761238622.53 total 467.96 last 23.19) - open/close 7710 (time 1761238635.55 total 480.98 last 12.75) - open/close 7924 (time 1761238646.36 total 491.79 last 19.81) - open/close 8159 (time 1761238656.36 total 501.79 last 23.49) - open/close 8302 (time 1761238666.71 total 512.14 last 13.81) - open/close 8484 (time 1761238676.72 total 522.15 last 18.19) - open/close 8679 (time 1761238687.11 total 532.55 last 18.76) - open/close 8918 (time 1761238697.12 total 542.55 last 23.89) - open/close 9126 (time 1761238708.19 total 553.62 last 18.79) - open/close 9332 (time 1761238718.28 total 563.72 last 20.41) - open/close 9498 (time 1761238728.85 total 574.29 last 15.70) - open/close 9682 (time 1761238738.87 total 584.30 last 18.38) - open/close 9877 (time 1761238749.45 total 594.88 last 18.42) - open/close 10000 (time 1761238756.59 total 602.02 last 17.23) - open/close 10226 (time 1761238768.85 total 614.28 last 18.44) - open/close 10365 (time 1761238779.66 total 625.09 last 12.86) - open/close 10487 (time 1761238789.67 total 635.11 last 12.18) - open/close 10711 (time 1761238799.70 total 645.14 last 22.33) - open/close 10910 (time 1761238810.24 total 655.68 last 18.88) - open/close 11104 (time 1761238820.25 total 665.68 last 19.39) - open/close 11311 (time 1761238830.25 total 675.69 last 20.69) - open/close 11509 (time 1761238840.40 total 685.83 last 19.52) - open/close 11717 (time 1761238850.62 total 696.06 last 20.34) - open/close 11969 (time 1761238860.88 total 706.32 last 24.56) - open/close 12178 (time 1761238871.23 total 716.67 last 20.19) - open/close 12378 (time 1761238881.24 total 726.68 last 19.98) - open/close 12632 (time 1761238891.91 total 737.34 last 23.81) - open/close 12840 (time 1761238901.93 total 747.36 last 20.76) - open/close 13024 (time 1761238912.72 total 758.15 last 17.05) - open/close 13233 (time 1761238922.74 total 768.17 last 20.86) - open/close 13435 (time 1761238932.75 total 778.18 last 20.19) - open/close 13646 (time 1761238942.99 total 788.42 last 20.59) - open/close 13832 (time 1761238952.99 total 798.43 last 18.60) - open/close 14049 (time 1761238963.00 total 808.44 last 21.68) - open/close 14238 (time 1761238973.01 total 818.44 last 18.89) - open/close 14471 (time 1761238983.01 total 828.45 last 23.29) - open/close 14661 (time 1761238993.02 total 838.45 last 18.99) - open/close 14823 (time 1761239003.17 total 848.60 last 15.96) - open/close 14987 (time 1761239013.18 total 858.61 last 16.38) - open/close 15111 (time 1761239023.78 total 869.21 last 11.70) - open/close 15306 (time 1761239033.78 total 879.21 last 19.50) - open/close 15504 (time 1761239043.80 total 889.23 last 19.76) - open/close 15664 (time 1761239053.81 total 899.24 last 15.98) - open/close 15841 (time 1761239064.88 total 910.31 last 15.99) - open/close 16014 (time 1761239077.41 total 922.84 last 13.81) - open/close 16528 (time 1761239087.42 total 932.86 last 51.33) - open/close 17081 (time 1761239097.42 total 942.86 last 55.29) - open/close 17704 (time 1761239107.45 total 952.88 last 62.17) - open/close 18332 (time 1761239117.45 total 962.88 last 62.80) - open/close 19028 (time 1761239127.46 total 972.89 last 69.53) - open/close 19721 (time 1761239137.46 total 982.89 last 69.28) - open/close 20000 (time 1761239141.78 total 987.21 last 64.59) - open/close 20704 (time 1761239151.78 total 997.22 last 70.37) - open/close 21347 (time 1761239161.79 total 1007.22 last 64.25) - open/close 21997 (time 1761239171.80 total 1017.23 last 64.95) - open/close 22661 (time 1761239181.80 total 1027.24 last 66.37) - open/close 23290 (time 1761239191.82 total 1037.25 last 62.83) - open/close 23987 (time 1761239201.82 total 1047.26 last 69.66) - open/close 24679 (time 1761239211.83 total 1057.26 last 69.17) - open/close 25332 (time 1761239221.84 total 1067.28 last 65.19) - open/close 26000 (time 1761239231.86 total 1077.30 last 66.67) - open/close 26668 (time 1761239241.88 total 1087.31 last 66.72) - open/close 27340 (time 1761239251.88 total 1097.31 last 67.16) - open/close 28042 (time 1761239261.88 total 1107.32 last 70.19) - open/close 28721 (time 1761239271.89 total 1117.32 last 67.89) - open/close 29346 (time 1761239281.89 total 1127.32 last 62.49) - open/close 30000 (time 1761239291.74 total 1137.17 last 66.37) - open/close 30686 (time 1761239301.75 total 1147.18 last 68.57) - open/close 31280 (time 1761239311.76 total 1157.19 last 59.31) - open/close 31925 (time 1761239321.77 total 1167.20 last 64.46) - open/close 32602 (time 1761239331.78 total 1177.22 last 67.59) - open/close 33253 (time 1761239341.79 total 1187.23 last 65.04) - open/close 33879 (time 1761239351.80 total 1197.23 last 62.55) - open/close 34531 (time 1761239361.81 total 1207.24 last 65.15) - open/close 35214 (time 1761239371.82 total 1217.25 last 68.24) - open/close 35845 (time 1761239381.84 total 1227.28 last 62.94) - open/close 36492 (time 1761239391.85 total 1237.28 last 64.68) - open/close 37106 (time 1761239401.85 total 1247.29 last 61.36) - open/close 37771 (time 1761239411.88 total 1257.31 last 66.35) - open/close 38380 (time 1761239421.88 total 1267.31 last 60.86) - open/close 38991 (time 1761239431.89 total 1277.33 last 61.03) - open/close 39664 (time 1761239441.90 total 1287.33 last 67.24) - open/close 40000 (time 1761239446.64 total 1292.08 last 70.84) - open/close 40696 (time 1761239456.64 total 1302.08 last 69.60) - open/close 41357 (time 1761239466.64 total 1312.08 last 66.09) - open/close 42002 (time 1761239476.66 total 1322.10 last 64.39) - open/close 42643 (time 1761239486.67 total 1332.10 last 64.06) - open/close 43280 (time 1761239496.68 total 1342.11 last 63.64) - open/close 43968 (time 1761239506.69 total 1352.12 last 68.71) - open/close 44641 (time 1761239516.69 total 1362.13 last 67.29) - open/close 45340 (time 1761239526.70 total 1372.13 last 69.88) - open/close 46046 (time 1761239536.70 total 1382.14 last 70.55) - open/close 46659 (time 1761239546.71 total 1392.14 last 61.26) - open/close 47330 (time 1761239556.72 total 1402.15 last 67.07) - open/close 47996 (time 1761239566.74 total 1412.17 last 66.44) - open/close 48662 (time 1761239576.74 total 1422.17 last 66.59) - open/close 49317 (time 1761239586.76 total 1432.19 last 65.38) - open/close 49988 (time 1761239596.77 total 1442.21 last 67.00) total: 50000 open/close in 1442.31 seconds: 34.67 ops/second - unlinked 0 (time 1761239601 ; total 0 ; last 0) - unlinked 10000 (time 1761239755 ; total 154 ; last 154) - unlinked 20000 (time 1761239905 ; total 304 ; last 150) - unlinked 30000 (time 1761240048 ; total 447 ; last 143) - unlinked 40000 (time 1761240194 ; total 593 ; last 146) total: 50000 unlinks in 730 seconds: 68.493149 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg254-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2