== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:45:14 (1780994714) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 235 (time 1780994811.21 total 10.01 last 23.47) - open/close 497 (time 1780994821.21 total 20.01 last 26.20) - open/close 732 (time 1780994831.22 total 30.01 last 23.50) - open/close 1003 (time 1780994842.11 total 40.91 last 24.88) - open/close 1208 (time 1780994854.80 total 53.60 last 16.15) - open/close 1406 (time 1780994864.80 total 63.60 last 19.79) - open/close 1619 (time 1780994875.01 total 73.81 last 20.87) - open/close 1889 (time 1780994885.84 total 84.64 last 24.93) - open/close 2149 (time 1780994896.07 total 94.87 last 25.42) - open/close 2352 (time 1780994906.08 total 104.88 last 20.28) - open/close 2636 (time 1780994916.82 total 115.61 last 26.46) - open/close 2850 (time 1780994926.83 total 125.63 last 21.37) - open/close 3113 (time 1780994937.22 total 136.02 last 25.31) - open/close 3376 (time 1780994948.10 total 146.89 last 24.18) - open/close 3606 (time 1780994958.10 total 156.90 last 22.99) - open/close 3835 (time 1780994968.45 total 167.25 last 22.12) - open/close 4046 (time 1780994978.89 total 177.69 last 20.22) - open/close 4282 (time 1780994988.89 total 187.69 last 23.59) - open/close 4521 (time 1780994999.31 total 198.10 last 22.95) - open/close 4775 (time 1780995010.06 total 208.86 last 23.62) - open/close 4997 (time 1780995020.08 total 218.87 last 22.16) - open/close 5203 (time 1780995030.81 total 229.61 last 19.19) - open/close 5379 (time 1780995041.73 total 240.52 last 16.13) - open/close 5484 (time 1780995051.74 total 250.54 last 10.48) - open/close 5691 (time 1780995061.76 total 260.55 last 20.67) - open/close 5935 (time 1780995071.76 total 270.56 last 24.39) - open/close 6164 (time 1780995081.77 total 280.57 last 22.87) - open/close 6395 (time 1780995091.79 total 290.59 last 23.07) - open/close 6644 (time 1780995102.78 total 301.58 last 22.65) - open/close 6887 (time 1780995112.79 total 311.59 last 24.27) - open/close 7159 (time 1780995122.81 total 321.60 last 27.16) - open/close 7379 (time 1780995132.82 total 331.61 last 21.98) - open/close 7535 (time 1780995142.82 total 341.62 last 15.59) - open/close 7772 (time 1780995153.73 total 352.53 last 21.72) - open/close 8007 (time 1780995163.93 total 362.72 last 23.06) - open/close 8250 (time 1780995174.44 total 373.24 last 23.12) - open/close 8460 (time 1780995184.45 total 383.24 last 20.98) - open/close 8704 (time 1780995194.99 total 393.79 last 23.14) - open/close 8946 (time 1780995205.52 total 404.31 last 22.99) - open/close 9229 (time 1780995216.47 total 415.27 last 25.84) - open/close 9528 (time 1780995226.48 total 425.28 last 29.85) - open/close 9682 (time 1780995236.76 total 435.56 last 14.99) - open/close 9935 (time 1780995247.25 total 446.04 last 24.12) - open/close 10000 (time 1780995249.51 total 448.31 last 28.71) - open/close 10196 (time 1780995259.53 total 458.33 last 19.57) - open/close 10470 (time 1780995270.35 total 469.15 last 25.31) - open/close 10598 (time 1780995281.12 total 479.92 last 11.89) - open/close 10805 (time 1780995291.43 total 490.23 last 20.07) - open/close 11034 (time 1780995302.47 total 501.26 last 20.75) - open/close 11308 (time 1780995313.30 total 512.09 last 25.30) - open/close 11548 (time 1780995323.59 total 522.39 last 23.32) - open/close 11765 (time 1780995334.42 total 533.22 last 20.03) - open/close 12001 (time 1780995345.22 total 544.02 last 21.85) - open/close 12178 (time 1780995355.22 total 554.02 last 17.70) - open/close 12380 (time 1780995365.23 total 564.03 last 20.19) - open/close 12640 (time 1780995375.36 total 574.16 last 25.65) - open/close 12887 (time 1780995386.11 total 584.91 last 22.98) - open/close 13135 (time 1780995396.86 total 595.66 last 23.06) - open/close 13341 (time 1780995406.93 total 605.73 last 20.47) - open/close 13547 (time 1780995417.82 total 616.62 last 18.91) - open/close 13801 (time 1780995428.19 total 626.99 last 24.51) - open/close 14011 (time 1780995438.34 total 637.13 last 20.69) - open/close 14193 (time 1780995449.24 total 648.03 last 16.70) - open/close 14316 (time 1780995459.24 total 658.03 last 12.30) - open/close 14412 (time 1780995469.71 total 668.50 last 9.17) - open/close 14679 (time 1780995480.41 total 679.21 last 24.94) - open/close 14900 (time 1780995490.62 total 689.42 last 21.65) - open/close 15122 (time 1780995500.63 total 699.43 last 22.17) - open/close 15364 (time 1780995511.04 total 709.84 last 23.25) - open/close 15699 (time 1780995521.05 total 719.84 last 33.48) - open/close 15929 (time 1780995531.05 total 729.84 last 23.00) - open/close 16139 (time 1780995541.07 total 739.87 last 20.95) - open/close 16350 (time 1780995551.08 total 749.87 last 21.09) - open/close 16528 (time 1780995561.78 total 760.58 last 16.62) - open/close 16779 (time 1780995572.35 total 771.15 last 23.76) - open/close 17007 (time 1780995582.36 total 781.16 last 22.77) - open/close 17187 (time 1780995593.11 total 791.91 last 16.75) - open/close 17475 (time 1780995603.11 total 801.91 last 28.78) - open/close 17680 (time 1780995613.12 total 811.91 last 20.50) - open/close 17764 (time 1780995623.72 total 822.51 last 7.92) - open/close 17942 (time 1780995633.72 total 832.52 last 17.79) - open/close 18216 (time 1780995643.74 total 842.54 last 27.36) - open/close 18391 (time 1780995653.76 total 852.56 last 17.46) - open/close 18644 (time 1780995664.54 total 863.33 last 23.48) - open/close 18891 (time 1780995675.24 total 874.04 last 23.08) - open/close 19149 (time 1780995685.70 total 884.50 last 24.65) - open/close 19339 (time 1780995696.32 total 895.12 last 17.89) - open/close 19510 (time 1780995706.33 total 905.12 last 17.09) - open/close 20000 (time 1780995714.43 total 913.23 last 60.44) - open/close 20804 (time 1780995724.44 total 923.24 last 80.34) - open/close 21551 (time 1780995734.44 total 933.24 last 74.68) - open/close 22299 (time 1780995744.45 total 943.25 last 74.72) - open/close 23037 (time 1780995754.46 total 953.25 last 73.77) - open/close 23845 (time 1780995764.46 total 963.26 last 80.74) - open/close 24645 (time 1780995774.47 total 973.27 last 79.93) - open/close 25463 (time 1780995784.48 total 983.27 last 81.78) - open/close 26233 (time 1780995794.49 total 993.28 last 76.92) - open/close 26930 (time 1780995804.50 total 1003.30 last 69.62) - open/close 27589 (time 1780995814.51 total 1013.31 last 65.82) - open/close 28337 (time 1780995824.51 total 1023.31 last 74.78) - open/close 29110 (time 1780995834.53 total 1033.32 last 77.19) - open/close 29927 (time 1780995844.53 total 1043.33 last 81.67) - open/close 30646 (time 1780995854.54 total 1053.34 last 71.83) - open/close 31108 (time 1780995864.54 total 1063.34 last 46.19) - open/close 31766 (time 1780995874.56 total 1073.36 last 65.69) - open/close 32476 (time 1780995884.56 total 1083.36 last 71.00) - open/close 33052 (time 1780995894.58 total 1093.37 last 57.52) - open/close 33685 (time 1780995904.58 total 1103.38 last 63.24) - open/close 34442 (time 1780995914.58 total 1113.38 last 75.69) - open/close 35098 (time 1780995924.64 total 1123.44 last 65.21) - open/close 35764 (time 1780995934.65 total 1133.45 last 66.56) - open/close 36370 (time 1780995944.66 total 1143.46 last 60.52) - open/close 37119 (time 1780995954.68 total 1153.47 last 74.81) - open/close 37790 (time 1780995964.68 total 1163.47 last 67.09) - open/close 38538 (time 1780995974.68 total 1173.48 last 74.74) - open/close 39261 (time 1780995984.70 total 1183.50 last 72.17) - open/close 40000 (time 1780995994.64 total 1193.44 last 74.36) - open/close 40708 (time 1780996004.64 total 1203.44 last 70.78) - open/close 41377 (time 1780996014.65 total 1213.45 last 66.86) - open/close 42050 (time 1780996024.67 total 1223.46 last 67.20) - open/close 42726 (time 1780996034.67 total 1233.47 last 67.53) - open/close 43423 (time 1780996044.68 total 1243.47 last 69.68) - open/close 44144 (time 1780996054.69 total 1253.48 last 72.03) - open/close 44857 (time 1780996064.69 total 1263.49 last 71.25) - open/close 45569 (time 1780996074.70 total 1273.49 last 71.19) - open/close 46313 (time 1780996084.71 total 1283.51 last 74.27) - open/close 47021 (time 1780996094.72 total 1293.51 last 70.77) - open/close 47700 (time 1780996104.73 total 1303.53 last 67.80) - open/close 48420 (time 1780996114.73 total 1313.53 last 71.99) - open/close 49154 (time 1780996124.74 total 1323.54 last 73.32) - open/close 49914 (time 1780996134.76 total 1333.56 last 75.89) total: 50000 open/close in 1334.74 seconds: 37.46 ops/second - unlinked 0 (time 1780996140 ; total 0 ; last 0) - unlinked 10000 (time 1780996286 ; total 146 ; last 146) - unlinked 20000 (time 1780996431 ; total 291 ; last 145) - unlinked 30000 (time 1780996574 ; total 434 ; last 143) - unlinked 40000 (time 1780996715 ; total 575 ; last 141) total: 50000 unlinks in 718 seconds: 69.637886 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2