== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:15:15 (1787858115) start mds service on oleg256-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg256-server: detected 4 online CPUs by sysfs oleg256-server: MODOPTS_LIBCFS= oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg256-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg256-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock 192.168.202.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 69 (time 1787858207.67 total 13.25 last 5.21) - open/close 343 (time 1787858217.68 total 23.26 last 27.36) - open/close 429 (time 1787858228.26 total 33.84 last 8.13) - open/close 550 (time 1787858238.70 total 44.28 last 11.58) - open/close 719 (time 1787858248.72 total 54.30 last 16.87) - open/close 964 (time 1787858259.64 total 65.22 last 22.44) - open/close 1066 (time 1787858270.01 total 75.59 last 9.84) - open/close 1348 (time 1787858280.54 total 86.12 last 26.76) - open/close 1556 (time 1787858291.56 total 97.14 last 18.88) - open/close 1715 (time 1787858302.45 total 108.03 last 14.60) - open/close 1924 (time 1787858312.82 total 118.40 last 20.16) - open/close 2098 (time 1787858323.06 total 128.64 last 16.98) - open/close 2148 (time 1787858333.90 total 139.48 last 4.61) - open/close 2217 (time 1787858343.91 total 149.49 last 6.89) - open/close 2284 (time 1787858354.73 total 160.31 last 6.19) - open/close 2471 (time 1787858364.76 total 170.34 last 18.65) - open/close 2666 (time 1787858375.55 total 181.13 last 18.07) - open/close 2760 (time 1787858385.58 total 191.16 last 9.37) - open/close 2839 (time 1787858395.74 total 201.32 last 7.78) - open/close 2954 (time 1787858405.74 total 211.32 last 11.49) - open/close 3143 (time 1787858416.24 total 221.82 last 18.00) - open/close 3292 (time 1787858426.43 total 232.01 last 14.62) - open/close 3488 (time 1787858436.66 total 242.23 last 19.17) - open/close 3696 (time 1787858446.68 total 252.26 last 20.75) - open/close 3792 (time 1787858458.09 total 263.67 last 8.41) - open/close 3962 (time 1787858468.71 total 274.29 last 16.00) - open/close 4091 (time 1787858479.46 total 285.04 last 12.00) - open/close 4314 (time 1787858489.49 total 295.07 last 22.25) - open/close 4434 (time 1787858499.50 total 305.08 last 11.99) - open/close 4549 (time 1787858510.47 total 316.05 last 10.48) - open/close 4724 (time 1787858520.49 total 326.07 last 17.47) - open/close 4915 (time 1787858530.52 total 336.10 last 19.04) - open/close 5081 (time 1787858540.97 total 346.55 last 15.88) - open/close 5268 (time 1787858554.31 total 359.89 last 14.02) - open/close 5392 (time 1787858564.49 total 370.07 last 12.18) - open/close 5482 (time 1787858574.49 total 380.07 last 8.99) - open/close 5663 (time 1787858586.48 total 392.06 last 15.10) - open/close 5837 (time 1787858596.51 total 402.09 last 17.35) - open/close 5948 (time 1787858609.61 total 415.19 last 8.47) - open/close 6165 (time 1787858619.61 total 425.19 last 21.68) - open/close 6326 (time 1787858629.62 total 435.20 last 16.09) - open/close 6501 (time 1787858639.63 total 445.20 last 17.49) - open/close 6709 (time 1787858649.97 total 455.55 last 20.11) - open/close 6875 (time 1787858660.50 total 466.08 last 15.76) - open/close 7140 (time 1787858670.79 total 476.37 last 25.77) - open/close 7403 (time 1787858681.72 total 487.30 last 24.04) - open/close 7575 (time 1787858692.62 total 498.20 last 15.78) - open/close 7747 (time 1787858702.64 total 508.22 last 17.17) - open/close 7893 (time 1787858714.43 total 520.01 last 12.38) - open/close 8049 (time 1787858724.45 total 530.03 last 15.57) - open/close 8299 (time 1787858734.64 total 540.22 last 24.54) - open/close 8486 (time 1787858745.14 total 550.72 last 17.81) - open/close 8594 (time 1787858755.16 total 560.74 last 10.78) - open/close 8744 (time 1787858765.34 total 570.92 last 14.73) - open/close 8967 (time 1787858775.35 total 580.93 last 22.28) - open/close 9137 (time 1787858785.37 total 590.95 last 16.98) - open/close 9287 (time 1787858796.40 total 601.98 last 13.59) - open/close 9527 (time 1787858807.22 total 612.80 last 22.18) - open/close 9756 (time 1787858817.76 total 623.34 last 21.74) - open/close 9845 (time 1787858829.30 total 634.88 last 7.71) - open/close 9919 (time 1787858839.35 total 644.93 last 7.37) - open/close 10000 (time 1787858842.99 total 648.57 last 22.26) - open/close 10249 (time 1787858853.04 total 658.62 last 24.77) - open/close 10408 (time 1787858865.90 total 671.48 last 12.36) - open/close 10610 (time 1787858875.92 total 681.50 last 20.17) - open/close 10797 (time 1787858886.89 total 692.47 last 17.05) - open/close 11011 (time 1787858896.89 total 702.47 last 21.39) - open/close 11249 (time 1787858909.17 total 714.75 last 19.38) - open/close 11454 (time 1787858920.17 total 725.75 last 18.63) - open/close 11480 (time 1787858934.13 total 739.71 last 1.86) - open/close 11675 (time 1787858944.15 total 749.73 last 19.47) - open/close 11790 (time 1787858954.99 total 760.57 last 10.60) - open/close 11988 (time 1787858965.31 total 770.89 last 19.20) - open/close 12216 (time 1787858975.32 total 780.90 last 22.77) - open/close 12366 (time 1787858985.68 total 791.26 last 14.48) - open/close 12550 (time 1787858995.94 total 801.52 last 17.93) - open/close 12620 (time 1787859006.29 total 811.87 last 6.77) - open/close 12734 (time 1787859017.00 total 822.58 last 10.64) - open/close 12935 (time 1787859027.04 total 832.62 last 20.01) - open/close 13160 (time 1787859037.08 total 842.66 last 22.42) - open/close 13334 (time 1787859047.09 total 852.67 last 17.39) - open/close 13427 (time 1787859057.13 total 862.71 last 9.26) - open/close 13660 (time 1787859067.13 total 872.71 last 23.29) - open/close 13797 (time 1787859077.29 total 882.87 last 13.49) - open/close 13957 (time 1787859087.30 total 892.88 last 15.98) - open/close 14124 (time 1787859097.72 total 903.30 last 16.03) - open/close 14248 (time 1787859107.73 total 913.31 last 12.39) - open/close 14366 (time 1787859118.30 total 923.88 last 11.15) - open/close 14510 (time 1787859128.33 total 933.91 last 14.36) - open/close 14639 (time 1787859138.94 total 944.52 last 12.16) - open/close 14824 (time 1787859149.22 total 954.80 last 17.99) - open/close 14898 (time 1787859159.98 total 965.56 last 6.88) - open/close 15014 (time 1787859170.21 total 975.79 last 11.34) - open/close 15122 (time 1787859180.52 total 986.10 last 10.47) - open/close 15356 (time 1787859191.59 total 997.17 last 21.14) - open/close 15521 (time 1787859202.47 total 1008.05 last 15.17) - open/close 15684 (time 1787859212.51 total 1018.08 last 16.25) - open/close 15805 (time 1787859222.71 total 1028.29 last 11.86) - open/close 15941 (time 1787859232.74 total 1038.32 last 13.56) - open/close 16101 (time 1787859243.63 total 1049.21 last 14.69) - open/close 16189 (time 1787859254.39 total 1059.97 last 8.18) - open/close 16363 (time 1787859265.25 total 1070.83 last 16.02) - open/close 16577 (time 1787859275.25 total 1080.83 last 21.39) - open/close 17388 (time 1787859285.26 total 1090.84 last 81.06) - open/close 18241 (time 1787859295.27 total 1100.85 last 85.22) - open/close 18831 (time 1787859305.28 total 1110.86 last 58.93) - open/close 19446 (time 1787859315.29 total 1120.87 last 61.47) - open/close 20000 (time 1787859323.32 total 1128.90 last 68.92) - open/close 20654 (time 1787859333.33 total 1138.91 last 65.34) - open/close 21339 (time 1787859343.34 total 1148.92 last 68.45) - open/close 22193 (time 1787859353.35 total 1158.93 last 85.37) - open/close 22771 (time 1787859363.36 total 1168.94 last 57.72) - open/close 23549 (time 1787859373.37 total 1178.95 last 77.76) - open/close 24222 (time 1787859383.37 total 1188.95 last 67.25) - open/close 24976 (time 1787859393.39 total 1198.97 last 75.30) - open/close 25667 (time 1787859403.39 total 1208.97 last 69.07) - open/close 26574 (time 1787859413.39 total 1218.97 last 90.70) - open/close 27374 (time 1787859423.39 total 1228.97 last 80.00) - open/close 28275 (time 1787859433.39 total 1238.97 last 90.08) - open/close 29084 (time 1787859443.40 total 1248.98 last 80.85) - open/close 29887 (time 1787859453.40 total 1258.98 last 80.29) - open/close 30633 (time 1787859463.41 total 1268.99 last 74.52) - open/close 31268 (time 1787859473.43 total 1279.01 last 63.39) - open/close 31939 (time 1787859483.44 total 1289.02 last 67.04) - open/close 32616 (time 1787859493.44 total 1299.02 last 67.67) - open/close 33308 (time 1787859503.45 total 1309.03 last 69.12) - open/close 33985 (time 1787859513.50 total 1319.08 last 67.40) - open/close 34755 (time 1787859523.51 total 1329.08 last 76.93) - open/close 35453 (time 1787859533.51 total 1339.09 last 69.80) - open/close 36217 (time 1787859543.51 total 1349.09 last 76.38) - open/close 37034 (time 1787859553.51 total 1359.09 last 81.70) - open/close 37894 (time 1787859563.52 total 1369.10 last 85.93) - open/close 38706 (time 1787859573.52 total 1379.10 last 81.18) - open/close 39587 (time 1787859583.52 total 1389.10 last 88.06) - open/close 40000 (time 1787859588.33 total 1393.91 last 85.99) - open/close 40752 (time 1787859598.33 total 1403.91 last 75.17) - open/close 41637 (time 1787859608.34 total 1413.92 last 88.41) - open/close 42366 (time 1787859618.34 total 1423.92 last 72.90) - open/close 43101 (time 1787859628.34 total 1433.92 last 73.47) - open/close 43893 (time 1787859638.35 total 1443.93 last 79.13) - open/close 44627 (time 1787859648.36 total 1453.94 last 73.37) - open/close 45525 (time 1787859658.36 total 1463.94 last 89.77) - open/close 46375 (time 1787859668.37 total 1473.95 last 84.95) - open/close 47156 (time 1787859678.37 total 1483.95 last 78.08) - open/close 47976 (time 1787859688.38 total 1493.96 last 81.91) - open/close 48814 (time 1787859698.39 total 1503.96 last 83.76) - open/close 49397 (time 1787859708.40 total 1513.98 last 58.23) total: 50000 open/close in 1521.97 seconds: 32.85 ops/second - unlinked 0 (time 1787859721 ; total 0 ; last 0) - unlinked 10000 (time 1787859861 ; total 140 ; last 140) - unlinked 20000 (time 1787859992 ; total 271 ; last 131) - unlinked 30000 (time 1787860118 ; total 397 ; last 126) - unlinked 40000 (time 1787860256 ; total 535 ; last 138) total: 50000 unlinks in 665 seconds: 75.187973 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server unloading modules via unload_modules_local on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local oleg256-server: modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2