== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:37:05 (1787035025) start mds service on oleg215-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 227 (time 1787035117.05 total 10.02 last 22.66) - open/close 470 (time 1787035127.07 total 20.03 last 24.26) - open/close 717 (time 1787035137.07 total 30.04 last 24.69) - open/close 855 (time 1787035147.55 total 40.52 last 13.17) - open/close 1046 (time 1787035158.32 total 51.29 last 17.74) - open/close 1255 (time 1787035168.55 total 61.51 last 20.43) - open/close 1407 (time 1787035178.56 total 71.53 last 15.18) - open/close 1575 (time 1787035189.21 total 82.17 last 15.78) - open/close 1718 (time 1787035199.26 total 92.23 last 14.22) - open/close 1944 (time 1787035209.30 total 102.27 last 22.51) - open/close 2134 (time 1787035220.45 total 113.41 last 17.05) - open/close 2362 (time 1787035230.46 total 123.43 last 22.77) - open/close 2592 (time 1787035243.50 total 136.47 last 17.64) - open/close 2793 (time 1787035254.55 total 147.51 last 18.20) - open/close 3011 (time 1787035265.12 total 158.09 last 20.61) - open/close 3265 (time 1787035275.13 total 168.10 last 25.37) - open/close 3428 (time 1787035285.93 total 178.90 last 15.09) - open/close 3623 (time 1787035298.09 total 191.06 last 16.04) - open/close 3747 (time 1787035308.83 total 201.79 last 11.55) - open/close 3986 (time 1787035318.83 total 211.80 last 23.88) - open/close 4134 (time 1787035328.85 total 221.81 last 14.78) - open/close 4291 (time 1787035338.86 total 231.82 last 15.69) - open/close 4502 (time 1787035349.68 total 242.64 last 19.50) - open/close 4741 (time 1787035360.10 total 253.06 last 22.94) - open/close 4919 (time 1787035370.94 total 263.90 last 16.41) - open/close 5017 (time 1787035382.11 total 275.07 last 8.78) - open/close 5269 (time 1787035392.11 total 285.07 last 25.20) - open/close 5494 (time 1787035402.11 total 295.08 last 22.49) - open/close 5735 (time 1787035412.64 total 305.61 last 22.88) - open/close 5977 (time 1787035431.73 total 324.70 last 12.68) - open/close 6169 (time 1787035442.10 total 335.06 last 18.52) - open/close 6349 (time 1787035452.11 total 345.07 last 17.98) - open/close 6478 (time 1787035462.13 total 355.09 last 12.87) - open/close 6708 (time 1787035473.04 total 366.01 last 21.07) - open/close 6902 (time 1787035483.26 total 376.23 last 18.99) - open/close 7147 (time 1787035494.26 total 387.22 last 22.28) - open/close 7382 (time 1787035504.26 total 397.22 last 23.50) - open/close 7611 (time 1787035514.26 total 407.22 last 22.90) - open/close 7763 (time 1787035524.46 total 417.42 last 14.90) - open/close 7986 (time 1787035534.47 total 427.43 last 22.28) - open/close 8144 (time 1787035545.00 total 437.97 last 14.99) - open/close 8288 (time 1787035555.00 total 447.97 last 14.40) - open/close 8437 (time 1787035565.01 total 457.98 last 14.89) - open/close 8619 (time 1787035575.39 total 468.36 last 17.53) - open/close 8872 (time 1787035585.43 total 478.39 last 25.21) - open/close 9095 (time 1787035595.43 total 488.40 last 22.29) - open/close 9310 (time 1787035605.44 total 498.41 last 21.48) - open/close 9496 (time 1787035615.45 total 508.42 last 18.58) - open/close 9738 (time 1787035625.46 total 518.43 last 24.18) - open/close 9948 (time 1787035636.02 total 528.99 last 19.89) - open/close 10000 (time 1787035639.42 total 532.39 last 15.28) - open/close 10200 (time 1787035649.96 total 542.92 last 18.99) - open/close 10341 (time 1787035660.09 total 553.05 last 13.91) - open/close 10572 (time 1787035670.12 total 563.09 last 23.03) - open/close 10739 (time 1787035680.13 total 573.09 last 16.69) - open/close 10845 (time 1787035691.06 total 584.03 last 9.69) - open/close 10997 (time 1787035702.69 total 595.66 last 13.07) - open/close 11174 (time 1787035712.70 total 605.67 last 17.68) - open/close 11293 (time 1787035724.25 total 617.22 last 10.30) - open/close 11475 (time 1787035734.26 total 627.22 last 18.18) - open/close 11619 (time 1787035744.36 total 637.33 last 14.25) - open/close 11820 (time 1787035754.38 total 647.35 last 20.06) - open/close 11930 (time 1787035765.33 total 658.30 last 10.04) - open/close 12166 (time 1787035775.61 total 668.57 last 22.97) - open/close 12348 (time 1787035785.88 total 678.85 last 17.71) - open/close 12491 (time 1787035795.91 total 688.88 last 14.26) - open/close 12621 (time 1787035806.26 total 699.22 last 12.56) - open/close 12817 (time 1787035816.99 total 709.95 last 18.26) - open/close 12959 (time 1787035827.75 total 720.71 last 13.20) - open/close 13081 (time 1787035837.93 total 730.89 last 11.98) - open/close 13140 (time 1787035847.93 total 740.90 last 5.90) - open/close 13271 (time 1787035857.95 total 750.92 last 13.07) - open/close 13490 (time 1787035868.45 total 761.42 last 20.86) - open/close 13707 (time 1787035878.79 total 771.75 last 21.00) - open/close 13919 (time 1787035889.12 total 782.09 last 20.51) - open/close 14206 (time 1787035899.91 total 792.87 last 26.60) - open/close 14469 (time 1787035909.93 total 802.90 last 26.24) - open/close 14715 (time 1787035921.02 total 813.98 last 22.19) - open/close 14929 (time 1787035933.81 total 826.78 last 16.72) - open/close 15171 (time 1787035943.82 total 836.78 last 24.18) - open/close 15353 (time 1787035954.52 total 847.49 last 17.00) - open/close 15589 (time 1787035964.54 total 857.50 last 23.57) - open/close 15764 (time 1787035974.56 total 867.52 last 17.46) - open/close 16070 (time 1787035984.56 total 877.52 last 30.60) - open/close 16228 (time 1787035994.80 total 887.77 last 15.42) - open/close 16429 (time 1787036005.09 total 898.06 last 19.53) - open/close 16688 (time 1787036015.11 total 908.07 last 25.86) - open/close 16903 (time 1787036025.11 total 918.08 last 21.49) - open/close 17036 (time 1787036035.14 total 928.10 last 13.27) - open/close 17279 (time 1787036045.28 total 938.24 last 23.96) - open/close 17547 (time 1787036056.15 total 949.12 last 24.64) - open/close 17770 (time 1787036066.18 total 959.14 last 22.24) - open/close 18367 (time 1787036076.18 total 969.15 last 59.68) - open/close 19095 (time 1787036086.19 total 979.15 last 72.76) - open/close 19790 (time 1787036096.19 total 989.15 last 69.50) - open/close 20000 (time 1787036099.33 total 992.29 last 66.90) - open/close 20715 (time 1787036109.34 total 1002.30 last 71.43) - open/close 21420 (time 1787036119.35 total 1012.31 last 70.43) - open/close 22175 (time 1787036129.35 total 1022.32 last 75.47) - open/close 22923 (time 1787036139.36 total 1032.32 last 74.76) - open/close 23588 (time 1787036149.36 total 1042.33 last 66.45) - open/close 24357 (time 1787036159.38 total 1052.35 last 76.76) - open/close 25054 (time 1787036169.39 total 1062.35 last 69.64) - open/close 25778 (time 1787036179.40 total 1072.37 last 72.32) - open/close 26353 (time 1787036189.41 total 1082.37 last 57.48) - open/close 27037 (time 1787036199.41 total 1092.37 last 68.39) - open/close 27624 (time 1787036209.41 total 1102.37 last 58.70) - open/close 28245 (time 1787036219.41 total 1112.38 last 62.05) - open/close 28964 (time 1787036229.42 total 1122.39 last 71.86) - open/close 29728 (time 1787036239.42 total 1132.39 last 76.39) - open/close 30000 (time 1787036243.30 total 1136.27 last 70.12) - open/close 30710 (time 1787036253.31 total 1146.27 last 70.94) - open/close 31512 (time 1787036263.31 total 1156.28 last 80.16) - open/close 32262 (time 1787036273.33 total 1166.29 last 74.89) - open/close 32946 (time 1787036283.34 total 1176.31 last 68.32) - open/close 33641 (time 1787036293.36 total 1186.33 last 69.36) - open/close 34330 (time 1787036303.37 total 1196.33 last 68.84) - open/close 34997 (time 1787036313.37 total 1206.33 last 66.70) - open/close 35817 (time 1787036323.38 total 1216.35 last 81.92) - open/close 36537 (time 1787036333.38 total 1226.35 last 71.98) - open/close 37217 (time 1787036343.40 total 1236.36 last 67.91) - open/close 37841 (time 1787036353.44 total 1246.41 last 62.11) - open/close 38506 (time 1787036363.45 total 1256.41 last 66.47) - open/close 39189 (time 1787036373.46 total 1266.42 last 68.23) - open/close 39895 (time 1787036383.46 total 1276.43 last 70.56) - open/close 40543 (time 1787036393.47 total 1286.44 last 64.74) - open/close 41038 (time 1787036403.51 total 1296.47 last 49.34) - open/close 41649 (time 1787036413.52 total 1306.48 last 61.03) - open/close 42339 (time 1787036423.53 total 1316.50 last 68.92) - open/close 43059 (time 1787036433.54 total 1326.50 last 71.94) - open/close 43701 (time 1787036443.56 total 1336.52 last 64.09) - open/close 44409 (time 1787036453.57 total 1346.54 last 70.67) - open/close 44949 (time 1787036463.58 total 1356.54 last 53.99) - open/close 45434 (time 1787036473.58 total 1366.54 last 48.48) - open/close 46117 (time 1787036483.59 total 1376.56 last 68.22) - open/close 46684 (time 1787036493.62 total 1386.58 last 56.55) - open/close 47442 (time 1787036503.62 total 1396.59 last 75.76) - open/close 48200 (time 1787036513.63 total 1406.60 last 75.74) - open/close 48881 (time 1787036523.66 total 1416.62 last 67.92) - open/close 49594 (time 1787036533.67 total 1426.63 last 71.22) total: 50000 open/close in 1433.95 seconds: 34.87 ops/second - unlinked 0 (time 1787036547 ; total 0 ; last 0) - unlinked 10000 (time 1787036721 ; total 174 ; last 174) - unlinked 20000 (time 1787036902 ; total 355 ; last 181) - unlinked 30000 (time 1787037065 ; total 518 ; last 163) - unlinked 40000 (time 1787037225 ; total 678 ; last 160) total: 50000 unlinks in 837 seconds: 59.737156 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2