== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:40:32 (1779205232) start mds service on oleg656-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 117 (time 1779205325.23 total 10.41 last 11.24) - open/close 295 (time 1779205335.25 total 20.43 last 17.77) - open/close 454 (time 1779205345.25 total 30.43 last 15.89) - open/close 644 (time 1779205355.62 total 40.80 last 18.33) - open/close 745 (time 1779205366.04 total 51.22 last 9.69) - open/close 847 (time 1779205377.06 total 62.24 last 9.26) - open/close 953 (time 1779205387.15 total 72.33 last 10.50) - open/close 996 (time 1779205398.85 total 84.03 last 3.68) - open/close 1053 (time 1779205408.86 total 94.04 last 5.69) - open/close 1101 (time 1779205419.55 total 104.73 last 4.49) - open/close 1187 (time 1779205430.41 total 115.59 last 7.92) - open/close 1325 (time 1779205441.12 total 126.30 last 12.88) - open/close 1498 (time 1779205451.37 total 136.55 last 16.88) - open/close 1633 (time 1779205461.38 total 146.56 last 13.49) - open/close 1819 (time 1779205471.45 total 156.63 last 18.47) - open/close 1972 (time 1779205481.46 total 166.64 last 15.29) - open/close 2073 (time 1779205491.48 total 176.66 last 10.07) - open/close 2283 (time 1779205501.48 total 186.66 last 21.00) - open/close 2323 (time 1779205511.75 total 196.93 last 3.90) - open/close 2386 (time 1779205522.07 total 207.24 last 6.11) - open/close 2502 (time 1779205533.26 total 218.44 last 10.36) - open/close 2642 (time 1779205545.22 total 230.40 last 11.71) - open/close 2899 (time 1779205555.24 total 240.42 last 25.65) - open/close 2963 (time 1779205565.34 total 250.52 last 6.34) - open/close 3132 (time 1779205582.74 total 267.92 last 9.71) - open/close 3149 (time 1779205593.75 total 278.93 last 1.55) - open/close 3170 (time 1779205605.67 total 290.85 last 1.76) - open/close 3275 (time 1779205618.52 total 303.70 last 8.17) - open/close 3417 (time 1779205629.21 total 314.39 last 13.29) - open/close 3449 (time 1779205639.39 total 324.57 last 3.14) - open/close 3531 (time 1779205651.66 total 336.84 last 6.68) - open/close 3724 (time 1779205661.67 total 346.85 last 19.28) - open/close 3855 (time 1779205672.28 total 357.46 last 12.34) - open/close 3886 (time 1779205683.29 total 368.47 last 2.82) - open/close 3951 (time 1779205693.65 total 378.83 last 6.27) - open/close 4016 (time 1779205704.69 total 389.87 last 5.89) - open/close 4100 (time 1779205715.17 total 400.35 last 8.02) - open/close 4256 (time 1779205727.21 total 412.39 last 12.95) - open/close 4312 (time 1779205737.81 total 422.99 last 5.29) - open/close 4402 (time 1779205748.73 total 433.91 last 8.24) - open/close 4464 (time 1779205758.74 total 443.92 last 6.20) - open/close 4654 (time 1779205769.11 total 454.29 last 18.31) - open/close 4796 (time 1779205780.10 total 465.28 last 12.92) - open/close 4985 (time 1779205790.78 total 475.96 last 17.70) - open/close 5125 (time 1779205801.39 total 486.57 last 13.19) - open/close 5153 (time 1779205811.39 total 496.57 last 2.80) - open/close 5277 (time 1779205821.45 total 506.63 last 12.33) - open/close 5322 (time 1779205832.33 total 517.51 last 4.14) - open/close 5507 (time 1779205842.34 total 527.52 last 18.49) - open/close 5590 (time 1779205852.56 total 537.74 last 8.12) - open/close 5759 (time 1779205862.76 total 547.94 last 16.56) - open/close 5870 (time 1779205872.77 total 557.95 last 11.10) - open/close 5973 (time 1779205883.67 total 568.85 last 9.45) - open/close 6126 (time 1779205895.71 total 580.89 last 12.70) - open/close 6231 (time 1779205906.78 total 591.96 last 9.49) - open/close 6305 (time 1779205917.83 total 603.01 last 6.69) - open/close 6460 (time 1779205927.85 total 613.03 last 15.48) - open/close 6584 (time 1779205937.87 total 623.05 last 12.37) - open/close 6732 (time 1779205947.88 total 633.06 last 14.78) - open/close 6851 (time 1779205958.10 total 643.27 last 11.65) - open/close 7050 (time 1779205968.83 total 654.01 last 18.54) - open/close 7208 (time 1779205979.61 total 664.79 last 14.65) - open/close 7342 (time 1779205990.23 total 675.40 last 12.62) - open/close 7539 (time 1779206002.22 total 687.40 last 16.43) - open/close 7723 (time 1779206013.86 total 699.04 last 15.80) - open/close 7897 (time 1779206023.88 total 709.05 last 17.38) - open/close 8066 (time 1779206033.88 total 719.06 last 16.89) - open/close 8234 (time 1779206044.06 total 729.24 last 16.51) - open/close 8361 (time 1779206055.05 total 740.23 last 11.55) - open/close 8496 (time 1779206065.07 total 750.25 last 13.47) - open/close 8713 (time 1779206075.08 total 760.26 last 21.67) - open/close 8883 (time 1779206085.08 total 770.26 last 17.00) - open/close 8985 (time 1779206097.82 total 783.00 last 8.01) - open/close 9096 (time 1779206108.11 total 793.29 last 10.79) - open/close 9128 (time 1779206118.44 total 803.62 last 3.10) - open/close 9340 (time 1779206128.44 total 813.62 last 21.19) - open/close 9550 (time 1779206138.58 total 823.76 last 20.72) - open/close 9677 (time 1779206149.59 total 834.76 last 11.54) - open/close 9846 (time 1779206159.61 total 844.79 last 16.87) - open/close 9984 (time 1779206170.31 total 855.49 last 12.90) - open/close 10192 (time 1779206180.33 total 865.51 last 20.76) - open/close 10296 (time 1779206191.88 total 877.06 last 9.00) - open/close 10511 (time 1779206201.89 total 887.07 last 21.47) - open/close 10707 (time 1779206211.90 total 897.08 last 19.59) - open/close 10884 (time 1779206222.18 total 907.36 last 17.21) - open/close 11026 (time 1779206233.00 total 918.18 last 13.13) - open/close 11220 (time 1779206243.29 total 928.47 last 18.85) - open/close 11347 (time 1779206253.30 total 938.48 last 12.69) - open/close 11473 (time 1779206264.04 total 949.22 last 11.73) - open/close 11642 (time 1779206274.17 total 959.34 last 16.69) - open/close 11794 (time 1779206284.20 total 969.38 last 15.15) - open/close 11979 (time 1779206294.95 total 980.13 last 17.21) - open/close 12135 (time 1779206306.93 total 992.10 last 13.02) - open/close 12329 (time 1779206317.04 total 1002.22 last 19.17) - open/close 12484 (time 1779206327.07 total 1012.25 last 15.46) - open/close 12651 (time 1779206337.77 total 1022.95 last 15.61) - open/close 12787 (time 1779206348.37 total 1033.55 last 12.83) - open/close 12908 (time 1779206358.71 total 1043.89 last 11.71) - open/close 13010 (time 1779206368.72 total 1053.90 last 10.19) - open/close 13142 (time 1779206378.72 total 1063.90 last 13.20) - open/close 13303 (time 1779206388.76 total 1073.94 last 16.05) - open/close 13499 (time 1779206399.20 total 1084.38 last 18.77) - open/close 14011 (time 1779206409.21 total 1094.39 last 51.12) - open/close 14592 (time 1779206419.21 total 1104.39 last 58.08) - open/close 15198 (time 1779206429.23 total 1114.41 last 60.51) - open/close 15758 (time 1779206439.23 total 1124.41 last 55.98) - open/close 16326 (time 1779206449.24 total 1134.42 last 56.75) - open/close 16962 (time 1779206459.25 total 1144.43 last 63.53) - open/close 17582 (time 1779206469.26 total 1154.44 last 61.96) - open/close 18159 (time 1779206479.26 total 1164.44 last 57.67) - open/close 18774 (time 1779206489.28 total 1174.46 last 61.38) - open/close 19340 (time 1779206499.30 total 1184.48 last 56.52) - open/close 19954 (time 1779206509.30 total 1194.48 last 61.36) - open/close 20534 (time 1779206519.33 total 1204.51 last 57.88) - open/close 21140 (time 1779206529.34 total 1214.52 last 60.54) - open/close 21757 (time 1779206539.36 total 1224.54 last 61.54) - open/close 22350 (time 1779206549.38 total 1234.56 last 59.21) - open/close 22954 (time 1779206559.38 total 1244.56 last 60.40) - open/close 23532 (time 1779206569.38 total 1254.56 last 57.77) - open/close 24136 (time 1779206579.39 total 1264.57 last 60.38) - open/close 24641 (time 1779206589.39 total 1274.57 last 50.47) - open/close 25171 (time 1779206599.41 total 1284.59 last 52.91) - open/close 25734 (time 1779206609.42 total 1294.60 last 56.26) - open/close 26295 (time 1779206619.42 total 1304.60 last 56.09) - open/close 26892 (time 1779206629.43 total 1314.60 last 59.67) - open/close 27400 (time 1779206639.44 total 1324.61 last 50.75) - open/close 27942 (time 1779206649.44 total 1334.62 last 54.17) - open/close 28531 (time 1779206659.45 total 1344.63 last 58.85) - open/close 29008 (time 1779206669.48 total 1354.66 last 47.55) - open/close 29449 (time 1779206679.48 total 1364.66 last 44.09) - open/close 29881 (time 1779206689.50 total 1374.68 last 43.14) - open/close 30000 (time 1779206692.10 total 1377.28 last 45.63) - open/close 30476 (time 1779206702.10 total 1387.28 last 47.60) - open/close 31046 (time 1779206712.12 total 1397.30 last 56.90) - open/close 31603 (time 1779206722.13 total 1407.31 last 55.65) - open/close 32197 (time 1779206732.13 total 1417.31 last 59.39) - open/close 32718 (time 1779206742.14 total 1427.32 last 52.08) - open/close 33278 (time 1779206752.15 total 1437.33 last 55.93) - open/close 33841 (time 1779206762.17 total 1447.35 last 56.20) - open/close 34445 (time 1779206772.17 total 1457.35 last 60.36) - open/close 34993 (time 1779206782.22 total 1467.40 last 54.53) - open/close 35454 (time 1779206792.23 total 1477.41 last 46.05) - open/close 35925 (time 1779206802.25 total 1487.43 last 47.04) - open/close 36374 (time 1779206812.26 total 1497.44 last 44.83) - open/close 36869 (time 1779206822.26 total 1507.44 last 49.49) - open/close 37399 (time 1779206832.27 total 1517.45 last 52.99) - open/close 37940 (time 1779206842.27 total 1527.45 last 54.07) - open/close 38516 (time 1779206852.28 total 1537.45 last 57.58) - open/close 39071 (time 1779206862.31 total 1547.49 last 55.31) - open/close 39626 (time 1779206872.32 total 1557.50 last 55.43) - open/close 40000 (time 1779206879.22 total 1564.40 last 54.23) - open/close 40515 (time 1779206889.23 total 1574.41 last 51.46) - open/close 41075 (time 1779206899.24 total 1584.42 last 55.95) - open/close 41614 (time 1779206909.25 total 1594.43 last 53.85) - open/close 42170 (time 1779206919.25 total 1604.43 last 55.57) - open/close 42739 (time 1779206929.25 total 1614.43 last 56.89) - open/close 43306 (time 1779206939.25 total 1624.43 last 56.70) - open/close 43794 (time 1779206949.27 total 1634.45 last 48.71) - open/close 44327 (time 1779206959.28 total 1644.46 last 53.26) - open/close 44824 (time 1779206969.29 total 1654.47 last 49.66) - open/close 45395 (time 1779206979.30 total 1664.48 last 57.05) - open/close 45974 (time 1779206989.31 total 1674.49 last 57.85) - open/close 46718 (time 1779206999.32 total 1684.50 last 74.28) - open/close 47303 (time 1779207009.32 total 1694.50 last 58.49) - open/close 47900 (time 1779207019.32 total 1704.50 last 59.70) - open/close 48475 (time 1779207029.33 total 1714.51 last 57.45) - open/close 49038 (time 1779207039.33 total 1724.51 last 56.29) - open/close 49581 (time 1779207049.34 total 1734.52 last 54.28) total: 50000 open/close in 1741.73 seconds: 28.71 ops/second - unlinked 0 (time 1779207060 ; total 0 ; last 0) - unlinked 10000 (time 1779207242 ; total 182 ; last 182) - unlinked 20000 (time 1779207437 ; total 377 ; last 195) - unlinked 30000 (time 1779207634 ; total 574 ; last 197) - unlinked 40000 (time 1779207818 ; total 758 ; last 184) total: 50000 unlinks in 940 seconds: 53.191490 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: modules unloaded. pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2