== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:52:44 (1785790364) start mds service on oleg122-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing load_modules_local oleg122-server: Loading modules from /home/green/git/lustre-release/lustre oleg122-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg122-server: detected 4 online CPUs by sysfs oleg122-server: MODOPTS_LIBCFS= oleg122-server: Force libcfs to create 2 CPU partitions oleg122-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg122-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg122-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg122-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg122-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg122-server: oleg122-server.virtnet: executing set_default_debug -1 all pdsh@oleg122-client: oleg122-server: ssh exited with exit code 1 Started lustre-OST0000 oleg122-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg122-client.virtnet: -o user_xattr,flock 192.168.201.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 126 (time 1785790467.76 total 10.01 last 12.59) - open/close 235 (time 1785790477.76 total 20.01 last 10.90) - open/close 282 (time 1785790488.46 total 30.71 last 4.39) - open/close 393 (time 1785790498.95 total 41.20 last 10.58) - open/close 506 (time 1785790509.00 total 51.25 last 11.25) - open/close 602 (time 1785790519.20 total 61.45 last 9.41) - open/close 692 (time 1785790529.21 total 71.46 last 8.99) - open/close 803 (time 1785790539.21 total 81.46 last 11.10) - open/close 927 (time 1785790549.26 total 91.51 last 12.34) - open/close 994 (time 1785790561.49 total 103.74 last 5.48) - open/close 1128 (time 1785790571.50 total 113.75 last 13.39) - open/close 1226 (time 1785790583.05 total 125.30 last 8.48) - open/close 1308 (time 1785790593.61 total 135.86 last 7.76) - open/close 1367 (time 1785790603.61 total 145.86 last 5.90) - open/close 1438 (time 1785790614.62 total 156.86 last 6.45) - open/close 1540 (time 1785790624.63 total 166.88 last 10.18) - open/close 1589 (time 1785790635.40 total 177.64 last 4.55) - open/close 1785 (time 1785790646.37 total 188.62 last 17.87) - open/close 1915 (time 1785790657.36 total 199.61 last 11.82) - open/close 1952 (time 1785790667.68 total 209.93 last 3.59) - open/close 2121 (time 1785790677.90 total 220.15 last 16.53) - open/close 2350 (time 1785790688.21 total 230.45 last 22.22) - open/close 2505 (time 1785790698.21 total 240.46 last 15.49) - open/close 2617 (time 1785790709.15 total 251.39 last 10.24) - open/close 2743 (time 1785790719.63 total 261.87 last 12.02) - open/close 2904 (time 1785790729.63 total 271.88 last 16.09) - open/close 3056 (time 1785790740.00 total 282.25 last 14.67) - open/close 3245 (time 1785790750.00 total 292.25 last 18.89) - open/close 3345 (time 1785790760.09 total 302.33 last 9.92) - open/close 3477 (time 1785790770.09 total 312.34 last 13.19) - open/close 3633 (time 1785790780.30 total 322.54 last 15.29) - open/close 3714 (time 1785790790.99 total 333.24 last 7.57) - open/close 3837 (time 1785790801.01 total 343.25 last 12.28) - open/close 3917 (time 1785790811.29 total 353.54 last 7.78) - open/close 4015 (time 1785790822.03 total 364.28 last 9.12) - open/close 4213 (time 1785790832.04 total 374.29 last 19.78) - open/close 4282 (time 1785790842.05 total 384.30 last 6.89) - open/close 4373 (time 1785790852.83 total 395.08 last 8.44) - open/close 4495 (time 1785790863.44 total 405.69 last 11.50) - open/close 4656 (time 1785790873.80 total 416.05 last 15.53) - open/close 4746 (time 1785790885.27 total 427.52 last 7.85) - open/close 4800 (time 1785790896.27 total 438.52 last 4.91) - open/close 4867 (time 1785790906.27 total 448.52 last 6.70) - open/close 5033 (time 1785790917.02 total 459.27 last 15.44) - open/close 5253 (time 1785790927.05 total 469.30 last 21.93) - open/close 5346 (time 1785790937.49 total 479.74 last 8.91) - open/close 5417 (time 1785790948.36 total 490.61 last 6.53) - open/close 5564 (time 1785790959.19 total 501.44 last 13.58) - open/close 5633 (time 1785790969.19 total 511.44 last 6.90) - open/close 5754 (time 1785790979.19 total 521.44 last 12.10) - open/close 5812 (time 1785790989.23 total 531.48 last 5.78) - open/close 5941 (time 1785790999.24 total 541.49 last 12.89) - open/close 6100 (time 1785791009.26 total 551.51 last 15.87) - open/close 6207 (time 1785791019.71 total 561.96 last 10.23) - open/close 6315 (time 1785791030.22 total 572.47 last 10.28) - open/close 6461 (time 1785791040.80 total 583.05 last 13.80) - open/close 6554 (time 1785791051.09 total 593.34 last 9.04) - open/close 6653 (time 1785791061.54 total 603.78 last 9.48) - open/close 6723 (time 1785791071.79 total 614.04 last 6.82) - open/close 6963 (time 1785791082.00 total 624.24 last 23.53) - open/close 7117 (time 1785791092.39 total 634.63 last 14.82) - open/close 7168 (time 1785791103.19 total 645.44 last 4.72) - open/close 7413 (time 1785791113.35 total 655.60 last 24.11) - open/close 7556 (time 1785791124.40 total 666.65 last 12.94) - open/close 7686 (time 1785791134.41 total 676.65 last 12.99) - open/close 7906 (time 1785791144.44 total 686.69 last 21.92) - open/close 8042 (time 1785791154.66 total 696.91 last 13.31) - open/close 8190 (time 1785791164.88 total 707.12 last 14.48) - open/close 8317 (time 1785791175.52 total 717.77 last 11.93) - open/close 8425 (time 1785791185.99 total 728.23 last 10.32) - open/close 8536 (time 1785791196.06 total 738.31 last 11.02) - open/close 8652 (time 1785791206.51 total 748.75 last 11.11) - open/close 8792 (time 1785791217.49 total 759.74 last 12.74) - open/close 8882 (time 1785791227.50 total 769.75 last 8.99) - open/close 9121 (time 1785791237.92 total 780.17 last 22.94) - open/close 9329 (time 1785791250.59 total 792.84 last 16.42) - open/close 9464 (time 1785791260.98 total 803.22 last 12.99) - open/close 9574 (time 1785791270.99 total 813.24 last 10.98) - open/close 9671 (time 1785791281.64 total 823.89 last 9.11) - open/close 9812 (time 1785791291.67 total 833.92 last 14.05) - open/close 10000 (time 1785791301.08 total 843.33 last 19.99) - open/close 10209 (time 1785791311.10 total 853.34 last 20.87) - open/close 10345 (time 1785791321.71 total 863.96 last 12.81) - open/close 10505 (time 1785791331.73 total 873.97 last 15.97) - open/close 10592 (time 1785791342.54 total 884.79 last 8.05) - open/close 10666 (time 1785791353.05 total 895.30 last 7.04) - open/close 10787 (time 1785791364.02 total 906.27 last 11.03) - open/close 10941 (time 1785791374.94 total 917.19 last 14.10) - open/close 10997 (time 1785791385.81 total 928.06 last 5.15) - open/close 11060 (time 1785791395.93 total 938.18 last 6.22) - open/close 11217 (time 1785791406.11 total 948.36 last 15.43) - open/close 11333 (time 1785791416.58 total 958.83 last 11.07) - open/close 11444 (time 1785791426.69 total 968.94 last 10.98) - open/close 11654 (time 1785791436.71 total 978.95 last 20.97) - open/close 11826 (time 1785791446.71 total 988.96 last 17.19) - open/close 11905 (time 1785791456.91 total 999.16 last 7.75) - open/close 11971 (time 1785791466.91 total 1009.16 last 6.60) - open/close 12068 (time 1785791477.78 total 1020.02 last 8.93) - open/close 12246 (time 1785791487.82 total 1030.07 last 17.72) - open/close 12463 (time 1785791498.07 total 1040.31 last 21.18) - open/close 12549 (time 1785791508.38 total 1050.63 last 8.34) - open/close 12587 (time 1785791518.88 total 1061.13 last 3.62) - open/close 12738 (time 1785791528.88 total 1071.13 last 15.09) - open/close 12844 (time 1785791539.10 total 1081.35 last 10.37) - open/close 13021 (time 1785791549.90 total 1092.15 last 16.39) - open/close 13152 (time 1785791560.93 total 1103.17 last 11.88) - open/close 13221 (time 1785791571.87 total 1114.12 last 6.30) - open/close 13359 (time 1785791583.29 total 1125.54 last 12.09) - open/close 13468 (time 1785791593.83 total 1136.08 last 10.34) - open/close 13633 (time 1785791603.85 total 1146.09 last 16.48) - open/close 13787 (time 1785791613.87 total 1156.11 last 15.37) - open/close 13899 (time 1785791624.39 total 1166.64 last 10.64) - open/close 14027 (time 1785791636.06 total 1178.31 last 10.97) - open/close 14147 (time 1785791646.80 total 1189.05 last 11.17) - open/close 14316 (time 1785791656.81 total 1199.06 last 16.87) - open/close 14771 (time 1785791666.87 total 1209.12 last 45.24) - open/close 15157 (time 1785791676.87 total 1219.12 last 38.59) - open/close 15571 (time 1785791686.89 total 1229.14 last 41.33) - open/close 16068 (time 1785791696.89 total 1239.14 last 49.69) - open/close 16737 (time 1785791706.90 total 1249.15 last 66.84) - open/close 17256 (time 1785791716.91 total 1259.16 last 51.86) - open/close 17771 (time 1785791726.91 total 1269.16 last 51.50) - open/close 18259 (time 1785791736.91 total 1279.16 last 48.78) - open/close 18801 (time 1785791746.92 total 1289.16 last 54.19) - open/close 19212 (time 1785791756.97 total 1299.21 last 40.90) - open/close 19722 (time 1785791766.99 total 1309.24 last 50.87) - open/close 20000 (time 1785791772.28 total 1314.53 last 52.53) - open/close 20442 (time 1785791782.29 total 1324.54 last 44.17) - open/close 20896 (time 1785791792.33 total 1334.58 last 45.23) - open/close 21459 (time 1785791802.33 total 1344.58 last 56.29) - open/close 21937 (time 1785791812.36 total 1354.61 last 47.65) - open/close 22493 (time 1785791822.37 total 1364.62 last 55.53) - open/close 22969 (time 1785791832.38 total 1374.62 last 47.58) - open/close 23546 (time 1785791842.38 total 1384.63 last 57.68) - open/close 24115 (time 1785791852.40 total 1394.65 last 56.76) - open/close 24436 (time 1785791862.42 total 1404.67 last 32.05) - open/close 24669 (time 1785791872.45 total 1414.70 last 23.23) - open/close 24953 (time 1785791882.48 total 1424.73 last 28.31) - open/close 25428 (time 1785791892.51 total 1434.76 last 47.34) - open/close 25763 (time 1785791902.55 total 1444.80 last 33.37) - open/close 26191 (time 1785791912.58 total 1454.83 last 42.66) - open/close 26626 (time 1785791922.59 total 1464.84 last 43.48) - open/close 27157 (time 1785791932.59 total 1474.84 last 53.08) - open/close 27894 (time 1785791942.60 total 1484.85 last 73.62) - open/close 28468 (time 1785791952.61 total 1494.86 last 57.34) - open/close 28912 (time 1785791962.63 total 1504.88 last 44.31) - open/close 29758 (time 1785791972.65 total 1514.90 last 84.47) - open/close 30000 (time 1785791975.91 total 1518.16 last 74.24) - open/close 30740 (time 1785791985.92 total 1528.17 last 73.93) - open/close 31291 (time 1785791995.93 total 1538.18 last 55.01) - open/close 31729 (time 1785792005.94 total 1548.18 last 43.79) - open/close 32160 (time 1785792015.95 total 1558.20 last 43.05) - open/close 32708 (time 1785792025.96 total 1568.20 last 54.76) - open/close 33279 (time 1785792035.96 total 1578.21 last 57.05) - open/close 33752 (time 1785792045.98 total 1588.23 last 47.24) - open/close 34209 (time 1785792056.00 total 1598.25 last 45.59) - open/close 34665 (time 1785792066.04 total 1608.29 last 45.41) - open/close 34925 (time 1785792077.50 total 1619.75 last 22.69) - open/close 35171 (time 1785792087.51 total 1629.76 last 24.57) - open/close 35382 (time 1785792097.53 total 1639.77 last 21.08) - open/close 35693 (time 1785792107.54 total 1649.78 last 31.07) - open/close 36066 (time 1785792117.57 total 1659.82 last 37.16) - open/close 36693 (time 1785792127.58 total 1669.82 last 62.69) - open/close 37192 (time 1785792137.63 total 1679.87 last 49.65) - open/close 37575 (time 1785792147.63 total 1689.88 last 38.29) - open/close 37979 (time 1785792157.65 total 1699.90 last 40.29) - open/close 38575 (time 1785792167.66 total 1709.91 last 59.58) - open/close 39097 (time 1785792177.67 total 1719.92 last 52.16) - open/close 39667 (time 1785792187.68 total 1729.93 last 56.92) - open/close 40000 (time 1785792192.93 total 1735.18 last 63.44) - open/close 40618 (time 1785792202.93 total 1745.18 last 61.80) - open/close 41100 (time 1785792212.96 total 1755.20 last 48.08) - open/close 41625 (time 1785792222.97 total 1765.22 last 52.43) - open/close 42477 (time 1785792232.97 total 1775.22 last 85.19) - open/close 43095 (time 1785792242.98 total 1785.23 last 61.72) - open/close 43602 (time 1785792252.99 total 1795.24 last 50.67) - open/close 44046 (time 1785792262.99 total 1805.24 last 44.39) - open/close 44527 (time 1785792273.01 total 1815.25 last 48.04) - open/close 45044 (time 1785792283.01 total 1825.26 last 51.68) - open/close 45535 (time 1785792293.01 total 1835.26 last 49.09) - open/close 46109 (time 1785792303.02 total 1845.26 last 57.38) - open/close 46569 (time 1785792313.03 total 1855.28 last 45.93) - open/close 47275 (time 1785792323.03 total 1865.28 last 70.60) - open/close 47910 (time 1785792333.04 total 1875.28 last 63.46) - open/close 48368 (time 1785792343.04 total 1885.29 last 45.78) - open/close 48997 (time 1785792353.06 total 1895.31 last 62.78) - open/close 49430 (time 1785792363.07 total 1905.32 last 43.26) total: 50000 open/close in 1913.64 seconds: 26.13 ops/second - unlinked 0 (time 1785792377 ; total 0 ; last 0) - unlinked 10000 (time 1785792559 ; total 182 ; last 182) - unlinked 20000 (time 1785792797 ; total 420 ; last 238) - unlinked 30000 (time 1785792994 ; total 617 ; last 197) - unlinked 40000 (time 1785793200 ; total 823 ; last 206) total: 50000 unlinks in 1026 seconds: 48.732944 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg122-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg122-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg122-server stop mds service on oleg122-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg122-server unloading modules via unload_modules_local on: 'oleg122-server' oleg122-server: oleg122-server.virtnet: executing unload_modules_local oleg122-server: modules unloaded. pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-server: ssh exited with exit code 2 pdsh@oleg122-client: oleg122-client: ssh exited with exit code 2