== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:05:14 (1789128314) start mds service on oleg319-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg319-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg319-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg319-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 256 (time 1789128422.85 total 12.54 last 20.42) - open/close 522 (time 1789128433.46 total 23.15 last 25.06) - open/close 822 (time 1789128443.48 total 33.17 last 29.94) - open/close 1051 (time 1789128453.61 total 43.30 last 22.61) - open/close 1325 (time 1789128464.01 total 53.70 last 26.34) - open/close 1581 (time 1789128474.77 total 64.46 last 23.79) - open/close 1783 (time 1789128484.78 total 74.47 last 20.18) - open/close 2013 (time 1789128495.24 total 84.93 last 21.99) - open/close 2249 (time 1789128505.54 total 95.23 last 22.91) - open/close 2463 (time 1789128515.56 total 105.25 last 21.35) - open/close 2727 (time 1789128526.53 total 116.22 last 24.06) - open/close 2975 (time 1789128537.10 total 126.79 last 23.47) - open/close 3249 (time 1789128547.13 total 136.82 last 27.31) - open/close 3493 (time 1789128558.01 total 147.70 last 22.42) - open/close 3695 (time 1789128569.09 total 158.78 last 18.23) - open/close 3903 (time 1789128579.41 total 169.10 last 20.16) - open/close 4104 (time 1789128590.47 total 180.16 last 18.18) - open/close 4305 (time 1789128600.47 total 190.16 last 20.09) - open/close 4566 (time 1789128610.49 total 200.17 last 26.06) - open/close 4762 (time 1789128621.40 total 211.08 last 17.97) - open/close 4893 (time 1789128631.63 total 221.32 last 12.80) - open/close 5102 (time 1789128641.66 total 231.35 last 20.85) - open/close 5325 (time 1789128652.12 total 241.81 last 21.31) - open/close 5619 (time 1789128662.78 total 252.47 last 27.58) - open/close 5736 (time 1789128673.48 total 263.17 last 10.94) - open/close 5911 (time 1789128683.84 total 273.53 last 16.88) - open/close 6160 (time 1789128693.85 total 283.54 last 24.89) - open/close 6257 (time 1789128703.89 total 293.58 last 9.66) - open/close 6523 (time 1789128714.26 total 303.95 last 25.64) - open/close 6847 (time 1789128725.18 total 314.87 last 29.67) - open/close 7113 (time 1789128735.82 total 325.51 last 25.00) - open/close 7354 (time 1789128745.85 total 335.54 last 24.03) - open/close 7540 (time 1789128756.54 total 346.23 last 17.39) - open/close 7820 (time 1789128767.04 total 356.72 last 26.69) - open/close 8056 (time 1789128777.74 total 367.42 last 22.05) - open/close 8248 (time 1789128787.75 total 377.44 last 19.18) - open/close 8461 (time 1789128797.75 total 387.44 last 21.29) - open/close 8684 (time 1789128808.65 total 398.33 last 20.47) - open/close 8940 (time 1789128818.75 total 408.44 last 25.34) - open/close 9141 (time 1789128828.76 total 418.45 last 20.08) - open/close 9380 (time 1789128839.23 total 428.92 last 22.83) - open/close 9582 (time 1789128850.12 total 439.81 last 18.55) - open/close 9648 (time 1789128860.24 total 449.93 last 6.52) - open/close 9872 (time 1789128870.24 total 459.93 last 22.40) - open/close 10000 (time 1789128874.55 total 464.24 last 29.72) - open/close 10204 (time 1789128884.73 total 474.42 last 20.03) - open/close 10453 (time 1789128895.35 total 485.04 last 23.45) - open/close 10632 (time 1789128906.04 total 495.73 last 16.74) - open/close 10790 (time 1789128916.74 total 506.42 last 14.78) - open/close 10939 (time 1789128926.97 total 516.66 last 14.56) - open/close 11178 (time 1789128937.85 total 527.54 last 21.98) - open/close 11377 (time 1789128947.86 total 537.54 last 19.88) - open/close 11671 (time 1789128958.48 total 548.17 last 27.67) - open/close 11788 (time 1789128969.29 total 558.98 last 10.82) - open/close 11939 (time 1789128979.77 total 569.46 last 14.41) - open/close 12144 (time 1789128990.54 total 580.22 last 19.04) - open/close 12341 (time 1789129000.96 total 590.65 last 18.90) - open/close 12490 (time 1789129013.52 total 603.21 last 11.86) - open/close 12629 (time 1789129023.52 total 613.21 last 13.89) - open/close 12792 (time 1789129034.52 total 624.21 last 14.82) - open/close 12947 (time 1789129046.46 total 636.15 last 12.98) - open/close 13106 (time 1789129057.01 total 646.70 last 15.08) - open/close 13264 (time 1789129067.02 total 656.71 last 15.78) - open/close 13449 (time 1789129077.02 total 666.71 last 18.49) - open/close 13695 (time 1789129087.89 total 677.57 last 22.65) - open/close 13938 (time 1789129098.70 total 688.39 last 22.46) - open/close 14211 (time 1789129109.35 total 699.03 last 25.65) - open/close 14474 (time 1789129119.36 total 709.04 last 26.27) - open/close 14729 (time 1789129129.36 total 719.05 last 25.50) - open/close 14914 (time 1789129140.29 total 729.98 last 16.92) - open/close 15137 (time 1789129150.32 total 740.01 last 22.25) - open/close 15363 (time 1789129160.33 total 750.01 last 22.58) - open/close 15639 (time 1789129170.33 total 760.02 last 27.58) - open/close 15863 (time 1789129180.73 total 770.42 last 21.54) - open/close 16124 (time 1789129190.75 total 780.44 last 26.07) - open/close 16249 (time 1789129200.76 total 790.45 last 12.49) - open/close 16434 (time 1789129211.50 total 801.19 last 17.22) - open/close 16590 (time 1789129222.60 total 812.29 last 14.06) - open/close 16777 (time 1789129232.64 total 822.33 last 18.62) - open/close 17055 (time 1789129242.65 total 832.34 last 27.77) - open/close 17283 (time 1789129252.68 total 842.37 last 22.74) - open/close 17609 (time 1789129263.69 total 853.38 last 29.62) - open/close 17854 (time 1789129273.69 total 863.38 last 24.49) - open/close 18028 (time 1789129283.70 total 873.39 last 17.38) - open/close 18265 (time 1789129293.71 total 883.40 last 23.68) - open/close 18426 (time 1789129303.76 total 893.45 last 16.03) - open/close 18641 (time 1789129313.86 total 903.54 last 21.29) - open/close 18901 (time 1789129324.87 total 914.56 last 23.61) - open/close 19172 (time 1789129334.88 total 924.57 last 27.08) - open/close 19401 (time 1789129344.88 total 934.57 last 22.90) - open/close 19646 (time 1789129355.57 total 945.26 last 22.92) - open/close 19899 (time 1789129366.05 total 955.74 last 24.15) - open/close 20000 (time 1789129369.80 total 959.49 last 26.91) - open/close 20460 (time 1789129379.80 total 969.49 last 45.99) - open/close 21169 (time 1789129389.82 total 979.51 last 70.80) - open/close 21936 (time 1789129399.84 total 989.53 last 76.53) - open/close 22705 (time 1789129409.84 total 999.53 last 76.89) - open/close 23435 (time 1789129419.85 total 1009.53 last 72.97) - open/close 24172 (time 1789129429.85 total 1019.54 last 73.66) - open/close 24844 (time 1789129439.87 total 1029.56 last 67.05) - open/close 25458 (time 1789129449.87 total 1039.56 last 61.40) - open/close 26131 (time 1789129459.88 total 1049.57 last 67.26) - open/close 26777 (time 1789129469.89 total 1059.58 last 64.55) - open/close 27428 (time 1789129479.89 total 1069.58 last 65.06) - open/close 28127 (time 1789129489.89 total 1079.58 last 69.89) - open/close 28805 (time 1789129499.90 total 1089.59 last 67.75) - open/close 29504 (time 1789129509.91 total 1099.60 last 69.86) - open/close 30000 (time 1789129516.32 total 1106.01 last 77.33) - open/close 30702 (time 1789129526.32 total 1116.01 last 70.19) - open/close 31370 (time 1789129536.32 total 1126.01 last 66.80) - open/close 32113 (time 1789129546.33 total 1136.02 last 74.27) - open/close 32870 (time 1789129556.34 total 1146.03 last 75.61) - open/close 33619 (time 1789129566.35 total 1156.04 last 74.83) - open/close 34283 (time 1789129576.36 total 1166.05 last 66.34) - open/close 35035 (time 1789129586.38 total 1176.07 last 75.02) - open/close 35738 (time 1789129596.39 total 1186.08 last 70.22) - open/close 36407 (time 1789129606.41 total 1196.10 last 66.81) - open/close 37126 (time 1789129616.41 total 1206.10 last 71.89) - open/close 37817 (time 1789129626.43 total 1216.11 last 68.99) - open/close 38620 (time 1789129636.43 total 1226.12 last 80.26) - open/close 39396 (time 1789129646.43 total 1236.12 last 77.56) - open/close 40000 (time 1789129654.87 total 1244.56 last 71.56) - open/close 40763 (time 1789129664.88 total 1254.56 last 76.29) - open/close 41488 (time 1789129674.89 total 1264.58 last 72.42) - open/close 42168 (time 1789129684.90 total 1274.58 last 67.94) - open/close 42911 (time 1789129694.90 total 1284.59 last 74.29) - open/close 43544 (time 1789129704.92 total 1294.61 last 63.15) - open/close 44156 (time 1789129714.93 total 1304.62 last 61.12) - open/close 44947 (time 1789129724.94 total 1314.62 last 79.08) - open/close 45695 (time 1789129734.94 total 1324.63 last 74.79) - open/close 46427 (time 1789129744.94 total 1334.63 last 73.14) - open/close 47184 (time 1789129754.95 total 1344.64 last 75.65) - open/close 47908 (time 1789129764.96 total 1354.65 last 72.31) - open/close 48551 (time 1789129774.98 total 1364.66 last 64.22) - open/close 49330 (time 1789129784.98 total 1374.67 last 77.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1379.05 seconds: 35.99 ops/second - unlinked 0 (time 1789129794 ; total 0 ; last 0) - unlinked 10000 (time 1789129933 ; total 139 ; last 139) - unlinked 20000 (time 1789130070 ; total 276 ; last 137) - unlinked 30000 (time 1789130215 ; total 421 ; last 145) - unlinked 40000 (time 1789130355 ; total 561 ; last 140) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 695 seconds: 71.410072 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules via unload_modules_local on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local oleg319-server: modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2