== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:59:12 (1787194752) start mds service on oleg635-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing load_modules_local oleg635-server: Loading modules from /home/green/git/lustre-release/lustre oleg635-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg635-server: detected 4 online CPUs by sysfs oleg635-server: MODOPTS_LIBCFS= oleg635-server: Force libcfs to create 2 CPU partitions oleg635-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg635-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg635-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg635-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg635-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 264 (time 1787194847.43 total 10.02 last 26.35) - open/close 480 (time 1787194858.15 total 20.74 last 20.15) - open/close 749 (time 1787194868.60 total 31.19 last 25.74) - open/close 989 (time 1787194878.61 total 41.20 last 23.97) - open/close 1211 (time 1787194889.52 total 52.10 last 20.36) - open/close 1417 (time 1787194900.09 total 62.67 last 19.49) - open/close 1640 (time 1787194910.95 total 73.54 last 20.52) - open/close 1845 (time 1787194921.45 total 84.04 last 19.53) - open/close 2133 (time 1787194932.07 total 94.66 last 27.12) - open/close 2387 (time 1787194942.25 total 104.84 last 24.96) - open/close 2618 (time 1787194952.76 total 115.34 last 21.99) - open/close 2776 (time 1787194963.18 total 125.76 last 15.16) - open/close 2992 (time 1787194973.18 total 135.77 last 21.59) - open/close 3199 (time 1787194983.98 total 146.57 last 19.17) - open/close 3403 (time 1787194993.99 total 156.58 last 20.38) - open/close 3646 (time 1787195004.00 total 166.59 last 24.28) - open/close 3888 (time 1787195014.00 total 176.59 last 24.19) - open/close 4114 (time 1787195024.31 total 186.89 last 21.94) - open/close 4314 (time 1787195034.61 total 197.20 last 19.41) - open/close 4536 (time 1787195044.87 total 207.46 last 21.64) - open/close 4702 (time 1787195054.97 total 217.56 last 16.44) - open/close 4928 (time 1787195064.98 total 227.57 last 22.58) - open/close 5177 (time 1787195075.89 total 238.47 last 22.83) - open/close 5371 (time 1787195086.14 total 248.72 last 18.92) - open/close 5628 (time 1787195096.14 total 258.73 last 25.69) - open/close 5839 (time 1787195106.44 total 269.02 last 20.50) - open/close 6060 (time 1787195117.10 total 279.68 last 20.73) - open/close 6266 (time 1787195127.25 total 289.84 last 20.28) - open/close 6505 (time 1787195137.28 total 299.86 last 23.84) - open/close 6707 (time 1787195147.64 total 310.23 last 19.49) - open/close 6949 (time 1787195157.73 total 320.32 last 23.98) - open/close 7150 (time 1787195167.86 total 330.45 last 19.85) - open/close 7346 (time 1787195177.92 total 340.50 last 19.49) - open/close 7572 (time 1787195188.53 total 351.11 last 21.30) - open/close 7820 (time 1787195199.22 total 361.81 last 23.19) - open/close 8050 (time 1787195209.22 total 371.81 last 23.00) - open/close 8297 (time 1787195219.24 total 381.83 last 24.65) - open/close 8546 (time 1787195230.14 total 392.73 last 22.85) - open/close 8812 (time 1787195240.64 total 403.23 last 25.33) - open/close 9059 (time 1787195250.65 total 413.23 last 24.69) - open/close 9264 (time 1787195260.68 total 423.26 last 20.44) - open/close 9477 (time 1787195271.60 total 434.18 last 19.51) - open/close 9746 (time 1787195282.09 total 444.67 last 25.64) - open/close 9981 (time 1787195292.72 total 455.30 last 22.11) - open/close 10170 (time 1787195302.83 total 465.41 last 18.69) - open/close 10384 (time 1787195312.83 total 475.42 last 21.39) - open/close 10612 (time 1787195323.12 total 485.70 last 22.17) - open/close 10815 (time 1787195334.08 total 496.67 last 18.51) - open/close 10987 (time 1787195344.62 total 507.21 last 16.32) - open/close 11187 (time 1787195354.62 total 517.21 last 20.00) - open/close 11501 (time 1787195365.18 total 527.76 last 29.76) - open/close 11766 (time 1787195375.19 total 537.77 last 26.47) - open/close 12018 (time 1787195385.22 total 547.80 last 25.13) - open/close 12234 (time 1787195395.26 total 557.84 last 21.51) - open/close 12491 (time 1787195405.26 total 567.85 last 25.69) - open/close 12742 (time 1787195415.28 total 577.86 last 25.06) - open/close 13024 (time 1787195425.28 total 587.87 last 28.19) - open/close 13260 (time 1787195435.69 total 598.28 last 22.66) - open/close 13507 (time 1787195445.70 total 608.29 last 24.69) - open/close 13782 (time 1787195456.70 total 619.29 last 24.99) - open/close 14017 (time 1787195466.99 total 629.57 last 22.85) - open/close 14287 (time 1787195477.56 total 640.15 last 25.54) - open/close 14555 (time 1787195487.56 total 650.15 last 26.80) - open/close 14819 (time 1787195498.49 total 661.08 last 24.15) - open/close 15017 (time 1787195508.50 total 671.09 last 19.79) - open/close 15237 (time 1787195518.64 total 681.23 last 21.69) - open/close 15451 (time 1787195529.61 total 692.19 last 19.52) - open/close 15620 (time 1787195540.36 total 702.95 last 15.72) - open/close 15783 (time 1787195550.66 total 713.25 last 15.83) - open/close 16030 (time 1787195561.04 total 723.62 last 23.80) - open/close 16280 (time 1787195571.52 total 734.11 last 23.84) - open/close 16529 (time 1787195581.55 total 744.14 last 24.83) - open/close 16788 (time 1787195591.66 total 754.25 last 25.62) - open/close 17034 (time 1787195602.22 total 764.81 last 23.29) - open/close 17263 (time 1787195612.79 total 775.38 last 21.67) - open/close 17473 (time 1787195622.79 total 785.38 last 20.99) - open/close 17670 (time 1787195632.81 total 795.39 last 19.67) - open/close 17870 (time 1787195642.87 total 805.45 last 19.88) - open/close 18533 (time 1787195652.88 total 815.46 last 66.25) - open/close 19375 (time 1787195662.88 total 825.47 last 84.16) - open/close 20000 (time 1787195670.28 total 832.87 last 84.48) - open/close 20902 (time 1787195680.29 total 842.87 last 90.11) - open/close 21833 (time 1787195690.29 total 852.88 last 93.04) - open/close 22641 (time 1787195700.30 total 862.88 last 80.77) - open/close 23544 (time 1787195710.31 total 872.89 last 90.21) - open/close 24374 (time 1787195720.32 total 882.90 last 82.92) - open/close 25231 (time 1787195730.32 total 892.90 last 85.69) - open/close 26097 (time 1787195740.32 total 902.91 last 86.59) - open/close 26971 (time 1787195750.33 total 912.91 last 87.34) - open/close 27797 (time 1787195760.34 total 922.92 last 82.52) - open/close 28673 (time 1787195770.34 total 932.93 last 87.56) - open/close 29490 (time 1787195780.35 total 942.93 last 81.66) - open/close 30000 (time 1787195786.42 total 949.01 last 83.95) - open/close 30844 (time 1787195796.42 total 959.01 last 84.37) - open/close 31580 (time 1787195806.43 total 969.01 last 73.59) - open/close 32368 (time 1787195816.44 total 979.02 last 78.71) - open/close 33213 (time 1787195826.44 total 989.03 last 84.44) - open/close 34063 (time 1787195836.45 total 999.04 last 84.94) - open/close 34860 (time 1787195846.46 total 1009.05 last 79.65) - open/close 35668 (time 1787195856.46 total 1019.05 last 80.78) - open/close 36505 (time 1787195866.48 total 1029.07 last 83.55) - open/close 37340 (time 1787195876.48 total 1039.07 last 83.48) - open/close 38213 (time 1787195886.49 total 1049.08 last 87.25) - open/close 39096 (time 1787195896.49 total 1059.08 last 88.25) - open/close 39861 (time 1787195906.50 total 1069.08 last 76.49) - open/close 40655 (time 1787195916.51 total 1079.09 last 79.32) - open/close 41479 (time 1787195926.52 total 1089.10 last 82.30) - open/close 42265 (time 1787195936.53 total 1099.11 last 78.53) - open/close 43105 (time 1787195946.53 total 1109.12 last 83.95) - open/close 43912 (time 1787195956.54 total 1119.12 last 80.65) - open/close 44665 (time 1787195966.55 total 1129.13 last 75.24) - open/close 45385 (time 1787195976.55 total 1139.13 last 72.00) - open/close 46220 (time 1787195986.56 total 1149.14 last 83.41) - open/close 47041 (time 1787195996.58 total 1159.16 last 81.93) - open/close 47890 (time 1787196006.58 total 1169.17 last 84.88) - open/close 48665 (time 1787196016.59 total 1179.18 last 77.43) - open/close 49415 (time 1787196026.59 total 1189.18 last 74.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1192.04 seconds: 41.64 ops/second - unlinked 0 (time 1787196034 ; total 0 ; last 0) - unlinked 10000 (time 1787196157 ; total 123 ; last 123) - unlinked 20000 (time 1787196284 ; total 250 ; last 127) - unlinked 30000 (time 1787196414 ; total 380 ; last 130) - unlinked 40000 (time 1787196548 ; total 514 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 651 seconds: 76.238098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg635-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg635-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg635-server unloading modules via unload_modules_local on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing unload_modules_local oleg635-server: modules unloaded. pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2