== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:08:34 (1787605714) start mds service on oleg103-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions oleg103-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg103-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg103-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg103-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1787605847.01 total 10.02 last 11.98) - open/close 284 (time 1787605857.05 total 20.06 last 16.34) - open/close 469 (time 1787605867.50 total 30.51 last 17.69) - open/close 708 (time 1787605878.51 total 41.52 last 21.72) - open/close 921 (time 1787605888.53 total 51.54 last 21.24) - open/close 1153 (time 1787605898.55 total 61.56 last 23.15) - open/close 1368 (time 1787605909.43 total 72.44 last 19.78) - open/close 1461 (time 1787605920.06 total 83.07 last 8.75) - open/close 1605 (time 1787605930.56 total 93.57 last 13.71) - open/close 1660 (time 1787605940.61 total 103.62 last 5.48) - open/close 1785 (time 1787605950.89 total 113.90 last 12.15) - open/close 1908 (time 1787605960.90 total 123.91 last 12.29) - open/close 2069 (time 1787605971.04 total 134.05 last 15.87) - open/close 2266 (time 1787605981.20 total 144.21 last 19.40) - open/close 2502 (time 1787605991.20 total 154.21 last 23.59) - open/close 2748 (time 1787606001.21 total 164.22 last 24.57) - open/close 2901 (time 1787606011.75 total 174.76 last 14.52) - open/close 2954 (time 1787606026.95 total 189.96 last 3.49) - open/close 3092 (time 1787606036.95 total 199.96 last 13.80) - open/close 3212 (time 1787606047.47 total 210.48 last 11.41) - open/close 3403 (time 1787606058.42 total 221.43 last 17.44) - open/close 3564 (time 1787606068.44 total 231.45 last 16.08) - open/close 3744 (time 1787606078.45 total 241.46 last 17.98) - open/close 3855 (time 1787606088.47 total 251.48 last 11.08) - open/close 3992 (time 1787606099.32 total 262.32 last 12.63) - open/close 4063 (time 1787606109.48 total 272.49 last 6.99) - open/close 4213 (time 1787606120.14 total 283.15 last 14.07) - open/close 4331 (time 1787606130.16 total 293.17 last 11.78) - open/close 4496 (time 1787606141.25 total 304.26 last 14.88) - open/close 4611 (time 1787606151.26 total 314.26 last 11.49) - open/close 4752 (time 1787606161.83 total 324.84 last 13.34) - open/close 4911 (time 1787606171.85 total 334.86 last 15.87) - open/close 5123 (time 1787606182.51 total 345.52 last 19.87) - open/close 5357 (time 1787606192.52 total 355.53 last 23.39) - open/close 5537 (time 1787606202.53 total 365.54 last 17.98) - open/close 5688 (time 1787606212.91 total 375.92 last 14.55) - open/close 5843 (time 1787606222.92 total 385.93 last 15.49) - open/close 6040 (time 1787606232.93 total 395.94 last 19.69) - open/close 6248 (time 1787606243.50 total 406.51 last 19.68) - open/close 6450 (time 1787606254.76 total 417.77 last 17.93) - open/close 6628 (time 1787606264.78 total 427.79 last 17.76) - open/close 6869 (time 1787606274.79 total 437.80 last 24.08) - open/close 7038 (time 1787606284.80 total 447.81 last 16.88) - open/close 7204 (time 1787606294.80 total 457.81 last 16.60) - open/close 7396 (time 1787606304.84 total 467.85 last 19.12) - open/close 7581 (time 1787606314.93 total 477.94 last 18.34) - open/close 7674 (time 1787606325.43 total 488.44 last 8.86) - open/close 7837 (time 1787606335.66 total 498.67 last 15.93) - open/close 8046 (time 1787606345.98 total 508.99 last 20.25) - open/close 8213 (time 1787606355.98 total 518.99 last 16.70) - open/close 8317 (time 1787606366.80 total 529.81 last 9.61) - open/close 8493 (time 1787606376.81 total 539.82 last 17.59) - open/close 8729 (time 1787606386.81 total 549.82 last 23.60) - open/close 8913 (time 1787606397.77 total 560.78 last 16.79) - open/close 8993 (time 1787606408.65 total 571.66 last 7.35) - open/close 9169 (time 1787606419.08 total 582.09 last 16.88) - open/close 9357 (time 1787606430.53 total 593.54 last 16.42) - open/close 9536 (time 1787606440.53 total 603.54 last 17.90) - open/close 9684 (time 1787606450.54 total 613.55 last 14.79) - open/close 9827 (time 1787606460.56 total 623.57 last 14.28) - open/close 9907 (time 1787606471.29 total 634.30 last 7.45) - open/close 10000 (time 1787606476.38 total 639.39 last 18.28) - open/close 10134 (time 1787606486.59 total 649.60 last 13.12) - open/close 10390 (time 1787606501.71 total 664.72 last 16.92) - open/close 10588 (time 1787606512.27 total 675.28 last 18.76) - open/close 10689 (time 1787606523.15 total 686.16 last 9.28) - open/close 10792 (time 1787606533.16 total 696.17 last 10.29) - open/close 10905 (time 1787606543.16 total 706.17 last 11.30) - open/close 11041 (time 1787606553.64 total 716.65 last 12.98) - open/close 11276 (time 1787606564.20 total 727.21 last 22.25) - open/close 11425 (time 1787606576.44 total 739.45 last 12.17) - open/close 11655 (time 1787606587.39 total 750.40 last 21.00) - open/close 11833 (time 1787606597.40 total 760.41 last 17.79) - open/close 12042 (time 1787606607.43 total 770.44 last 20.84) - open/close 12321 (time 1787606618.48 total 781.49 last 25.26) - open/close 12582 (time 1787606629.17 total 792.18 last 24.40) - open/close 12796 (time 1787606639.34 total 802.35 last 21.04) - open/close 13081 (time 1787606649.72 total 812.73 last 27.45) - open/close 13252 (time 1787606660.62 total 823.63 last 15.70) - open/close 13360 (time 1787606670.65 total 833.66 last 10.77) - open/close 13595 (time 1787606681.26 total 844.27 last 22.15) - open/close 13847 (time 1787606691.66 total 854.67 last 24.22) - open/close 13943 (time 1787606701.67 total 864.68 last 9.59) - open/close 14158 (time 1787606711.70 total 874.71 last 21.44) - open/close 14376 (time 1787606722.17 total 885.18 last 20.81) - open/close 14594 (time 1787606732.91 total 895.92 last 20.30) - open/close 14660 (time 1787606743.54 total 906.55 last 6.21) - open/close 14967 (time 1787606753.54 total 916.55 last 30.69) - open/close 15099 (time 1787606763.56 total 926.57 last 13.19) - open/close 15329 (time 1787606773.56 total 936.57 last 23.00) - open/close 15546 (time 1787606783.92 total 946.93 last 20.93) - open/close 15680 (time 1787606794.67 total 957.68 last 12.47) - open/close 15870 (time 1787606805.63 total 968.64 last 17.33) - open/close 16065 (time 1787606815.65 total 978.66 last 19.47) - open/close 16272 (time 1787606825.79 total 988.80 last 20.41) - open/close 16424 (time 1787606835.80 total 998.81 last 15.19) - open/close 16609 (time 1787606847.53 total 1010.54 last 15.76) - open/close 16778 (time 1787606857.65 total 1020.66 last 16.70) - open/close 16972 (time 1787606867.68 total 1030.69 last 19.35) - open/close 17387 (time 1787606877.69 total 1040.70 last 41.44) - open/close 18202 (time 1787606887.70 total 1050.71 last 81.48) - open/close 19128 (time 1787606897.70 total 1060.71 last 92.54) - open/close 19866 (time 1787606907.70 total 1070.71 last 73.79) - open/close 20625 (time 1787606917.72 total 1080.73 last 75.81) - open/close 21398 (time 1787606927.75 total 1090.76 last 77.03) - open/close 22162 (time 1787606937.76 total 1100.77 last 76.36) - open/close 22868 (time 1787606947.77 total 1110.78 last 70.51) - open/close 23564 (time 1787606957.77 total 1120.78 last 69.59) - open/close 24444 (time 1787606967.79 total 1130.80 last 87.85) - open/close 25115 (time 1787606977.81 total 1140.82 last 66.94) - open/close 25654 (time 1787606987.81 total 1150.82 last 53.88) - open/close 26276 (time 1787606997.83 total 1160.84 last 62.11) - open/close 27011 (time 1787607007.86 total 1170.87 last 73.28) - open/close 27677 (time 1787607017.86 total 1180.87 last 66.58) - open/close 28518 (time 1787607027.88 total 1190.89 last 83.98) - open/close 29262 (time 1787607037.88 total 1200.89 last 74.38) - open/close 30000 (time 1787607047.78 total 1210.79 last 74.55) - open/close 30811 (time 1787607057.78 total 1220.79 last 81.06) - open/close 31607 (time 1787607067.79 total 1230.80 last 79.55) - open/close 32283 (time 1787607077.79 total 1240.80 last 67.58) - open/close 32972 (time 1787607087.81 total 1250.82 last 68.76) - open/close 33811 (time 1787607097.82 total 1260.83 last 83.83) - open/close 34544 (time 1787607107.83 total 1270.84 last 73.26) - open/close 35321 (time 1787607117.83 total 1280.84 last 77.68) - open/close 36065 (time 1787607127.84 total 1290.85 last 74.30) - open/close 36783 (time 1787607137.85 total 1300.86 last 71.73) - open/close 37513 (time 1787607147.86 total 1310.87 last 72.98) - open/close 38345 (time 1787607157.86 total 1320.87 last 83.13) - open/close 38980 (time 1787607167.87 total 1330.88 last 63.48) - open/close 39685 (time 1787607177.87 total 1340.88 last 70.50) - open/close 40000 (time 1787607181.51 total 1344.52 last 86.57) - open/close 40753 (time 1787607191.51 total 1354.52 last 75.27) - open/close 41346 (time 1787607201.51 total 1364.52 last 59.27) - open/close 41951 (time 1787607211.52 total 1374.53 last 60.48) - open/close 42691 (time 1787607221.52 total 1384.53 last 73.96) - open/close 43454 (time 1787607231.53 total 1394.54 last 76.23) - open/close 44259 (time 1787607241.54 total 1404.55 last 80.41) - open/close 44943 (time 1787607251.55 total 1414.56 last 68.32) - open/close 45636 (time 1787607261.56 total 1424.57 last 69.24) - open/close 46463 (time 1787607271.57 total 1434.58 last 82.63) - open/close 47234 (time 1787607281.58 total 1444.59 last 77.05) - open/close 47955 (time 1787607291.58 total 1454.59 last 72.07) - open/close 48723 (time 1787607301.61 total 1464.62 last 76.56) - open/close 49327 (time 1787607311.62 total 1474.63 last 60.33) - open/close 49960 (time 1787607321.63 total 1484.64 last 63.28) total: 50000 open/close in 1485.14 seconds: 33.67 ops/second - unlinked 0 (time 1787607326 ; total 0 ; last 0) - unlinked 10000 (time 1787607473 ; total 147 ; last 147) - unlinked 20000 (time 1787607635 ; total 309 ; last 162) - unlinked 30000 (time 1787607803 ; total 477 ; last 168) - unlinked 40000 (time 1787607970 ; total 644 ; last 167) total: 50000 unlinks in 785 seconds: 63.694267 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg103-server unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: modules unloaded. pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2