== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:39:44 (1786077584) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1786077675.32 total 11.00 last 17.73) - open/close 347 (time 1786077685.34 total 21.02 last 15.17) - open/close 509 (time 1786077695.34 total 31.02 last 16.20) - open/close 659 (time 1786077706.02 total 41.70 last 14.05) - open/close 803 (time 1786077716.58 total 52.25 last 13.64) - open/close 988 (time 1786077727.62 total 63.30 last 16.75) - open/close 1077 (time 1786077738.47 total 74.15 last 8.20) - open/close 1234 (time 1786077748.48 total 84.16 last 15.69) - open/close 1471 (time 1786077758.76 total 94.43 last 23.06) - open/close 1643 (time 1786077769.11 total 104.78 last 16.62) - open/close 1825 (time 1786077779.22 total 114.90 last 17.99) - open/close 2004 (time 1786077790.24 total 125.92 last 16.24) - open/close 2233 (time 1786077801.21 total 136.89 last 20.88) - open/close 2406 (time 1786077811.95 total 147.63 last 16.11) - open/close 2633 (time 1786077821.97 total 157.64 last 22.67) - open/close 2849 (time 1786077832.43 total 168.11 last 20.63) - open/close 3036 (time 1786077842.67 total 178.35 last 18.26) - open/close 3179 (time 1786077852.70 total 188.38 last 14.27) - open/close 3259 (time 1786077865.31 total 200.99 last 6.34) - open/close 3372 (time 1786077875.33 total 211.01 last 11.28) - open/close 3458 (time 1786077886.33 total 222.01 last 7.82) - open/close 3655 (time 1786077896.87 total 232.55 last 18.69) - open/close 3892 (time 1786077907.58 total 243.26 last 22.12) - open/close 4036 (time 1786077917.59 total 253.27 last 14.39) - open/close 4149 (time 1786077927.79 total 263.46 last 11.08) - open/close 4343 (time 1786077937.80 total 273.47 last 19.38) - open/close 4553 (time 1786077949.67 total 285.35 last 17.69) - open/close 4652 (time 1786077959.68 total 295.36 last 9.89) - open/close 4858 (time 1786077970.36 total 306.03 last 19.30) - open/close 5053 (time 1786077981.17 total 316.85 last 18.03) - open/close 5264 (time 1786077991.18 total 326.86 last 21.09) - open/close 5376 (time 1786078002.23 total 337.91 last 10.13) - open/close 5557 (time 1786078012.79 total 348.47 last 17.15) - open/close 5726 (time 1786078023.86 total 359.54 last 15.27) - open/close 5796 (time 1786078034.23 total 369.90 last 6.75) - open/close 5977 (time 1786078044.91 total 380.59 last 16.93) - open/close 6203 (time 1786078055.24 total 390.91 last 21.90) - open/close 6350 (time 1786078065.24 total 400.92 last 14.69) - open/close 6482 (time 1786078075.95 total 411.63 last 12.33) - open/close 6719 (time 1786078086.50 total 422.18 last 22.46) - open/close 6904 (time 1786078097.75 total 433.43 last 16.44) - open/close 7118 (time 1786078108.07 total 443.75 last 20.74) - open/close 7326 (time 1786078118.51 total 454.19 last 19.93) - open/close 7498 (time 1786078128.68 total 464.36 last 16.92) - open/close 7650 (time 1786078138.68 total 474.36 last 15.19) - open/close 7808 (time 1786078149.23 total 484.91 last 14.99) - open/close 8054 (time 1786078159.59 total 495.26 last 23.75) - open/close 8244 (time 1786078169.59 total 505.27 last 18.98) - open/close 8435 (time 1786078180.27 total 515.95 last 17.89) - open/close 8666 (time 1786078190.51 total 526.19 last 22.56) - open/close 8812 (time 1786078200.53 total 536.21 last 14.57) - open/close 8976 (time 1786078210.71 total 546.39 last 16.11) - open/close 9082 (time 1786078220.98 total 556.66 last 10.32) - open/close 9228 (time 1786078231.26 total 566.94 last 14.20) - open/close 9432 (time 1786078241.78 total 577.46 last 19.39) - open/close 9540 (time 1786078251.79 total 587.46 last 10.79) - open/close 9751 (time 1786078261.84 total 597.51 last 20.99) - open/close 9902 (time 1786078272.81 total 608.48 last 13.77) - open/close 10000 (time 1786078279.29 total 614.97 last 15.11) - open/close 10093 (time 1786078290.30 total 625.98 last 8.45) - open/close 10240 (time 1786078301.26 total 636.93 last 13.42) - open/close 10350 (time 1786078311.28 total 646.96 last 10.97) - open/close 10596 (time 1786078321.28 total 656.96 last 24.59) - open/close 10718 (time 1786078331.29 total 666.96 last 12.20) - open/close 10953 (time 1786078341.30 total 676.97 last 23.48) - open/close 11104 (time 1786078353.69 total 689.37 last 12.18) - open/close 11304 (time 1786078363.73 total 699.41 last 19.93) - open/close 11507 (time 1786078374.52 total 710.20 last 18.81) - open/close 11732 (time 1786078384.57 total 720.25 last 22.39) - open/close 11883 (time 1786078395.52 total 731.20 last 13.79) - open/close 12048 (time 1786078405.53 total 741.21 last 16.48) - open/close 12228 (time 1786078415.61 total 751.28 last 17.87) - open/close 12386 (time 1786078426.41 total 762.09 last 14.62) - open/close 12604 (time 1786078436.41 total 772.09 last 21.80) - open/close 12671 (time 1786078446.75 total 782.42 last 6.48) - open/close 12890 (time 1786078457.32 total 793.00 last 20.72) - open/close 13009 (time 1786078467.51 total 803.19 last 11.68) - open/close 13104 (time 1786078478.46 total 814.14 last 8.67) - open/close 13291 (time 1786078488.97 total 824.65 last 17.80) - open/close 13382 (time 1786078498.98 total 834.66 last 9.09) - open/close 13535 (time 1786078511.59 total 847.26 last 12.14) - open/close 13740 (time 1786078521.59 total 857.26 last 20.50) - open/close 13930 (time 1786078532.13 total 867.81 last 18.01) - open/close 14146 (time 1786078542.89 total 878.56 last 20.09) - open/close 14345 (time 1786078553.76 total 889.43 last 18.31) - open/close 14494 (time 1786078563.96 total 899.64 last 14.60) - open/close 14626 (time 1786078574.65 total 910.33 last 12.34) - open/close 14770 (time 1786078586.73 total 922.41 last 11.93) - open/close 14914 (time 1786078597.67 total 933.34 last 13.17) - open/close 15114 (time 1786078610.33 total 946.01 last 15.79) - open/close 15238 (time 1786078621.03 total 956.71 last 11.59) - open/close 15384 (time 1786078632.60 total 968.28 last 12.62) - open/close 15474 (time 1786078643.29 total 978.97 last 8.42) - open/close 15504 (time 1786078653.32 total 988.99 last 2.99) - open/close 15634 (time 1786078663.82 total 999.50 last 12.38) - open/close 15729 (time 1786078674.80 total 1010.48 last 8.65) - open/close 15948 (time 1786078685.47 total 1021.14 last 20.54) - open/close 16152 (time 1786078695.72 total 1031.39 last 19.90) - open/close 16376 (time 1786078705.98 total 1041.66 last 21.83) - open/close 16566 (time 1786078715.99 total 1051.67 last 18.98) - open/close 16701 (time 1786078726.86 total 1062.54 last 12.42) - open/close 16855 (time 1786078737.71 total 1073.38 last 14.19) - open/close 17083 (time 1786078748.32 total 1084.00 last 21.48) - open/close 17234 (time 1786078758.52 total 1094.19 last 14.81) - open/close 17682 (time 1786078768.52 total 1104.20 last 44.79) - open/close 18277 (time 1786078778.54 total 1114.21 last 59.40) - open/close 18852 (time 1786078788.54 total 1124.21 last 57.50) - open/close 19475 (time 1786078798.54 total 1134.22 last 62.26) - open/close 20000 (time 1786078807.41 total 1143.09 last 59.17) - open/close 20618 (time 1786078817.42 total 1153.10 last 61.74) - open/close 21270 (time 1786078827.44 total 1163.12 last 65.08) - open/close 21888 (time 1786078837.46 total 1173.13 last 61.71) - open/close 22508 (time 1786078847.47 total 1183.14 last 61.94) - open/close 23161 (time 1786078857.48 total 1193.15 last 65.23) - open/close 23720 (time 1786078867.49 total 1203.16 last 55.84) - open/close 24280 (time 1786078877.50 total 1213.18 last 55.92) - open/close 24854 (time 1786078887.52 total 1223.20 last 57.30) - open/close 25461 (time 1786078897.53 total 1233.21 last 60.64) - open/close 26122 (time 1786078907.53 total 1243.21 last 66.08) - open/close 26752 (time 1786078917.54 total 1253.22 last 62.95) - open/close 27372 (time 1786078927.54 total 1263.22 last 61.96) - open/close 28021 (time 1786078937.55 total 1273.23 last 64.86) - open/close 28679 (time 1786078947.56 total 1283.23 last 65.77) - open/close 29315 (time 1786078957.58 total 1293.25 last 63.47) - open/close 29927 (time 1786078967.58 total 1303.26 last 61.18) - open/close 30494 (time 1786078977.59 total 1313.27 last 56.63) - open/close 31097 (time 1786078987.60 total 1323.28 last 60.23) - open/close 31667 (time 1786078997.60 total 1333.28 last 57.00) - open/close 32218 (time 1786079007.60 total 1343.28 last 55.10) - open/close 32765 (time 1786079017.61 total 1353.29 last 54.65) - open/close 33307 (time 1786079027.63 total 1363.31 last 54.11) - open/close 33844 (time 1786079037.64 total 1373.32 last 53.63) - open/close 34430 (time 1786079047.64 total 1383.32 last 58.58) - open/close 34948 (time 1786079057.66 total 1393.33 last 51.74) - open/close 35529 (time 1786079067.67 total 1403.34 last 58.05) - open/close 36090 (time 1786079077.68 total 1413.36 last 56.01) - open/close 36575 (time 1786079087.69 total 1423.37 last 48.46) - open/close 37084 (time 1786079097.69 total 1433.37 last 50.89) - open/close 37677 (time 1786079107.70 total 1443.38 last 59.25) - open/close 38293 (time 1786079117.71 total 1453.39 last 61.56) - open/close 38897 (time 1786079127.72 total 1463.39 last 60.35) - open/close 39510 (time 1786079137.72 total 1473.40 last 61.25) - open/close 40000 (time 1786079145.71 total 1481.39 last 61.35) - open/close 40571 (time 1786079155.72 total 1491.39 last 57.07) - open/close 41135 (time 1786079165.73 total 1501.41 last 56.33) - open/close 41673 (time 1786079175.73 total 1511.41 last 53.79) - open/close 42186 (time 1786079185.73 total 1521.41 last 51.28) - open/close 42795 (time 1786079195.74 total 1531.41 last 60.88) - open/close 43350 (time 1786079205.74 total 1541.41 last 55.50) - open/close 43938 (time 1786079215.74 total 1551.42 last 58.75) - open/close 44608 (time 1786079225.75 total 1561.43 last 66.98) - open/close 45204 (time 1786079235.75 total 1571.43 last 59.56) - open/close 45821 (time 1786079245.77 total 1581.44 last 61.63) - open/close 46419 (time 1786079255.77 total 1591.45 last 59.75) - open/close 46989 (time 1786079265.79 total 1601.47 last 56.91) - open/close 47554 (time 1786079275.81 total 1611.48 last 56.41) - open/close 48082 (time 1786079285.82 total 1621.49 last 52.74) - open/close 48640 (time 1786079295.83 total 1631.51 last 55.72) - open/close 49188 (time 1786079305.83 total 1641.51 last 54.79) - open/close 49722 (time 1786079315.84 total 1651.51 last 53.38) total: 50000 open/close in 1656.71 seconds: 30.18 ops/second - unlinked 0 (time 1786079326 ; total 0 ; last 0) - unlinked 10000 (time 1786079523 ; total 197 ; last 197) - unlinked 20000 (time 1786079727 ; total 401 ; last 204) - unlinked 30000 (time 1786079925 ; total 599 ; last 198) - unlinked 40000 (time 1786080126 ; total 800 ; last 201) total: 50000 unlinks in 986 seconds: 50.709938 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2