== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:06:13 (1781229973) start mds service on oleg458-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing load_modules_local oleg458-server: Loading modules from /home/green/git/lustre-release/lustre oleg458-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg458-server: detected 4 online CPUs by sysfs oleg458-server: MODOPTS_LIBCFS= oleg458-server: Force libcfs to create 2 CPU partitions oleg458-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg458-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg458-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg458-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1 Started lustre-OST0000 oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg458-client.virtnet: -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1781230067.47 total 10.80 last 13.34) - open/close 211 (time 1781230078.43 total 21.76 last 6.11) - open/close 438 (time 1781230091.02 total 34.35 last 18.03) - open/close 602 (time 1781230101.25 total 44.58 last 16.04) - open/close 877 (time 1781230111.95 total 55.28 last 25.70) - open/close 1069 (time 1781230122.26 total 65.59 last 18.62) - open/close 1164 (time 1781230132.94 total 76.26 last 8.90) - open/close 1339 (time 1781230142.95 total 86.28 last 17.48) - open/close 1433 (time 1781230153.25 total 96.58 last 9.12) - open/close 1501 (time 1781230164.10 total 107.43 last 6.27) - open/close 1583 (time 1781230174.55 total 117.88 last 7.85) - open/close 1744 (time 1781230184.58 total 127.91 last 16.05) - open/close 1891 (time 1781230194.59 total 137.91 last 14.69) - open/close 2010 (time 1781230204.60 total 147.93 last 11.89) - open/close 2142 (time 1781230219.86 total 163.19 last 8.65) - open/close 2301 (time 1781230230.54 total 173.87 last 14.88) - open/close 2392 (time 1781230241.05 total 184.38 last 8.66) - open/close 2605 (time 1781230251.79 total 195.12 last 19.83) - open/close 2731 (time 1781230262.02 total 205.35 last 12.32) - open/close 2920 (time 1781230272.40 total 215.73 last 18.22) - open/close 3073 (time 1781230282.82 total 226.15 last 14.68) - open/close 3192 (time 1781230292.82 total 236.15 last 11.90) - open/close 3383 (time 1781230302.87 total 246.20 last 19.01) - open/close 3394 (time 1781230315.40 total 258.73 last 0.88) - open/close 3611 (time 1781230325.58 total 268.91 last 21.32) - open/close 3685 (time 1781230335.59 total 278.91 last 7.39) - open/close 3779 (time 1781230346.65 total 289.97 last 8.50) - open/close 3944 (time 1781230356.66 total 299.99 last 16.48) - open/close 4062 (time 1781230366.68 total 310.01 last 11.77) - open/close 4227 (time 1781230377.41 total 320.74 last 15.38) - open/close 4385 (time 1781230388.24 total 331.57 last 14.59) - open/close 4563 (time 1781230398.26 total 341.59 last 17.77) - open/close 4793 (time 1781230408.64 total 351.97 last 22.15) - open/close 5013 (time 1781230418.68 total 362.01 last 21.92) - open/close 5229 (time 1781230429.38 total 372.70 last 20.20) - open/close 5418 (time 1781230439.57 total 382.90 last 18.53) - open/close 5582 (time 1781230452.06 total 395.39 last 13.14) - open/close 5743 (time 1781230462.08 total 405.41 last 16.06) - open/close 5936 (time 1781230472.33 total 415.65 last 18.84) - open/close 6012 (time 1781230482.34 total 425.67 last 7.59) - open/close 6204 (time 1781230493.37 total 436.70 last 17.40) - open/close 6408 (time 1781230503.69 total 447.02 last 19.78) - open/close 6467 (time 1781230514.70 total 458.02 last 5.36) - open/close 6549 (time 1781230524.97 total 468.29 last 7.98) - open/close 6743 (time 1781230535.33 total 478.66 last 18.71) - open/close 6817 (time 1781230545.36 total 488.68 last 7.38) - open/close 6914 (time 1781230555.66 total 498.99 last 9.41) - open/close 7041 (time 1781230565.71 total 509.04 last 12.64) - open/close 7167 (time 1781230575.73 total 519.05 last 12.58) - open/close 7282 (time 1781230586.45 total 529.78 last 10.73) - open/close 7340 (time 1781230599.44 total 542.77 last 4.46) - open/close 7458 (time 1781230609.44 total 552.77 last 11.80) - open/close 7630 (time 1781230619.49 total 562.81 last 17.12) - open/close 7806 (time 1781230629.89 total 573.22 last 16.92) - open/close 8031 (time 1781230639.90 total 583.23 last 22.47) - open/close 8218 (time 1781230650.31 total 593.64 last 17.97) - open/close 8455 (time 1781230660.33 total 603.66 last 23.65) - open/close 8654 (time 1781230671.32 total 614.64 last 18.11) - open/close 8895 (time 1781230681.66 total 624.99 last 23.29) - open/close 9053 (time 1781230691.79 total 635.11 last 15.61) - open/close 9224 (time 1781230702.40 total 645.73 last 16.11) - open/close 9331 (time 1781230712.41 total 655.74 last 10.69) - open/close 9536 (time 1781230722.83 total 666.16 last 19.66) - open/close 9654 (time 1781230733.91 total 677.24 last 10.66) - open/close 9815 (time 1781230743.93 total 687.25 last 16.07) - open/close 10000 (time 1781230752.43 total 695.76 last 21.76) - open/close 10255 (time 1781230762.44 total 705.77 last 25.47) - open/close 10402 (time 1781230772.77 total 716.10 last 14.23) - open/close 10669 (time 1781230783.22 total 726.54 last 25.56) - open/close 10844 (time 1781230793.23 total 736.56 last 17.47) - open/close 11039 (time 1781230803.40 total 746.73 last 19.18) - open/close 11256 (time 1781230813.57 total 756.90 last 21.33) - open/close 11517 (time 1781230823.89 total 767.22 last 25.30) - open/close 11722 (time 1781230833.89 total 777.22 last 20.50) - open/close 11829 (time 1781230844.70 total 788.03 last 9.90) - open/close 11978 (time 1781230854.70 total 798.03 last 14.89) - open/close 12153 (time 1781230865.17 total 808.49 last 16.73) - open/close 12396 (time 1781230875.19 total 818.52 last 24.24) - open/close 12646 (time 1781230885.20 total 828.53 last 24.96) - open/close 12800 (time 1781230895.82 total 839.14 last 14.51) - open/close 12974 (time 1781230905.82 total 849.15 last 17.39) - open/close 13193 (time 1781230916.18 total 859.51 last 21.14) - open/close 13442 (time 1781230926.81 total 870.14 last 23.42) - open/close 13705 (time 1781230937.80 total 881.13 last 23.93) - open/close 13895 (time 1781230948.46 total 891.79 last 17.83) - open/close 14127 (time 1781230958.47 total 901.80 last 23.18) - open/close 14379 (time 1781230968.48 total 911.80 last 25.18) - open/close 14654 (time 1781230979.53 total 922.85 last 24.89) - open/close 14863 (time 1781230990.85 total 934.18 last 18.45) - open/close 15122 (time 1781231000.91 total 944.24 last 25.76) - open/close 15347 (time 1781231010.92 total 954.25 last 22.46) - open/close 15526 (time 1781231020.92 total 964.25 last 17.90) - open/close 15646 (time 1781231030.98 total 974.31 last 11.94) - open/close 15873 (time 1781231040.99 total 984.32 last 22.68) - open/close 16051 (time 1781231050.99 total 994.32 last 17.79) - open/close 16249 (time 1781231061.20 total 1004.52 last 19.40) - open/close 16479 (time 1781231072.10 total 1015.43 last 21.09) - open/close 16672 (time 1781231082.27 total 1025.59 last 18.99) - open/close 16888 (time 1781231093.11 total 1036.44 last 19.91) - open/close 17601 (time 1781231103.12 total 1046.45 last 71.25) - open/close 18293 (time 1781231113.14 total 1056.47 last 69.09) - open/close 18963 (time 1781231123.15 total 1066.48 last 66.91) - open/close 19598 (time 1781231133.16 total 1076.48 last 63.46) - open/close 20000 (time 1781231138.57 total 1081.90 last 74.21) - open/close 20723 (time 1781231148.58 total 1091.91 last 72.22) - open/close 21411 (time 1781231158.59 total 1101.92 last 68.76) - open/close 22114 (time 1781231168.59 total 1111.92 last 70.27) - open/close 22828 (time 1781231178.60 total 1121.93 last 71.35) - open/close 23522 (time 1781231188.60 total 1131.93 last 69.39) - open/close 24261 (time 1781231198.61 total 1141.94 last 73.83) - open/close 24869 (time 1781231208.62 total 1151.94 last 60.77) - open/close 25578 (time 1781231218.62 total 1161.94 last 70.89) - open/close 26241 (time 1781231228.62 total 1171.95 last 66.27) - open/close 26836 (time 1781231238.64 total 1181.96 last 59.42) - open/close 27418 (time 1781231248.64 total 1191.97 last 58.18) - open/close 27958 (time 1781231258.65 total 1201.98 last 53.95) - open/close 28542 (time 1781231268.66 total 1211.98 last 58.35) - open/close 29168 (time 1781231278.67 total 1222.00 last 62.50) - open/close 29797 (time 1781231288.68 total 1232.00 last 62.88) - open/close 30000 (time 1781231291.71 total 1235.04 last 66.89) - open/close 30689 (time 1781231301.72 total 1245.05 last 68.84) - open/close 31419 (time 1781231311.74 total 1255.07 last 72.86) - open/close 32086 (time 1781231321.74 total 1265.07 last 66.66) - open/close 32763 (time 1781231331.75 total 1275.08 last 67.65) - open/close 33409 (time 1781231341.76 total 1285.09 last 64.54) - open/close 34109 (time 1781231351.77 total 1295.10 last 69.91) - open/close 34755 (time 1781231361.78 total 1305.10 last 64.58) - open/close 35423 (time 1781231371.78 total 1315.11 last 66.75) - open/close 36034 (time 1781231381.80 total 1325.13 last 60.97) - open/close 36708 (time 1781231391.80 total 1335.13 last 67.40) - open/close 37392 (time 1781231401.81 total 1345.13 last 68.39) - open/close 38026 (time 1781231411.82 total 1355.15 last 63.33) - open/close 38620 (time 1781231421.82 total 1365.15 last 59.39) - open/close 39261 (time 1781231431.84 total 1375.16 last 63.99) - open/close 39927 (time 1781231441.84 total 1385.17 last 66.57) - open/close 40560 (time 1781231451.85 total 1395.17 last 63.27) - open/close 41166 (time 1781231461.86 total 1405.19 last 60.49) - open/close 41650 (time 1781231471.86 total 1415.19 last 48.40) - open/close 42287 (time 1781231481.87 total 1425.19 last 63.69) - open/close 42863 (time 1781231491.88 total 1435.21 last 57.51) - open/close 43479 (time 1781231501.89 total 1445.22 last 61.54) - open/close 43955 (time 1781231511.90 total 1455.22 last 47.58) - open/close 44548 (time 1781231521.90 total 1465.22 last 59.30) - open/close 45182 (time 1781231531.90 total 1475.23 last 63.39) - open/close 45792 (time 1781231541.91 total 1485.24 last 60.93) - open/close 46474 (time 1781231551.92 total 1495.25 last 68.11) - open/close 47142 (time 1781231561.93 total 1505.25 last 66.77) - open/close 47757 (time 1781231571.94 total 1515.27 last 61.41) - open/close 48402 (time 1781231581.95 total 1525.28 last 64.44) - open/close 49039 (time 1781231591.96 total 1535.29 last 63.62) - open/close 49610 (time 1781231601.97 total 1545.30 last 57.05) total: 50000 open/close in 1551.79 seconds: 32.22 ops/second - unlinked 0 (time 1781231612 ; total 0 ; last 0) - unlinked 10000 (time 1781231776 ; total 164 ; last 164) - unlinked 20000 (time 1781231936 ; total 324 ; last 160) - unlinked 30000 (time 1781232094 ; total 482 ; last 158) - unlinked 40000 (time 1781232259 ; total 647 ; last 165) total: 50000 unlinks in 823 seconds: 60.753342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg458-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg458-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server stop mds service on oleg458-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server unloading modules via unload_modules_local on: 'oleg458-server' oleg458-server: oleg458-server.virtnet: executing unload_modules_local oleg458-server: modules unloaded. pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2 pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2