== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:53:24 (1787147604) start mds service on oleg136-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg136-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg136-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1787147701.77 total 10.25 last 10.24) - open/close 256 (time 1787147712.02 total 20.50 last 14.73) - open/close 432 (time 1787147722.03 total 30.51 last 17.58) - open/close 630 (time 1787147732.05 total 40.53 last 19.77) - open/close 773 (time 1787147742.78 total 51.25 last 13.33) - open/close 956 (time 1787147754.63 total 63.11 last 15.44) - open/close 1038 (time 1787147764.64 total 73.12 last 8.19) - open/close 1290 (time 1787147775.13 total 83.60 last 24.03) - open/close 1568 (time 1787147785.14 total 93.62 last 27.75) - open/close 1833 (time 1787147795.14 total 103.62 last 26.50) - open/close 1930 (time 1787147805.17 total 113.65 last 9.68) - open/close 2110 (time 1787147815.17 total 123.65 last 18.00) - open/close 2410 (time 1787147826.25 total 134.72 last 27.08) - open/close 2559 (time 1787147836.31 total 144.79 last 14.80) - open/close 2778 (time 1787147846.33 total 154.81 last 21.86) - open/close 3048 (time 1787147857.35 total 165.83 last 24.50) - open/close 3288 (time 1787147867.36 total 175.84 last 23.97) - open/close 3577 (time 1787147878.43 total 186.91 last 26.11) - open/close 3859 (time 1787147889.47 total 197.95 last 25.54) - open/close 4123 (time 1787147899.49 total 207.97 last 26.35) - open/close 4374 (time 1787147909.78 total 218.26 last 24.40) - open/close 4645 (time 1787147919.78 total 228.26 last 27.08) - open/close 4984 (time 1787147930.58 total 239.06 last 31.39) - open/close 5238 (time 1787147941.51 total 249.99 last 23.24) - open/close 5546 (time 1787147951.52 total 260.00 last 30.77) - open/close 5816 (time 1787147961.63 total 270.10 last 26.73) - open/close 6071 (time 1787147972.22 total 280.70 last 24.07) - open/close 6290 (time 1787147982.53 total 291.01 last 21.23) - open/close 6583 (time 1787147995.34 total 303.82 last 22.87) - open/close 6867 (time 1787148005.58 total 314.05 last 27.76) - open/close 7072 (time 1787148016.26 total 324.74 last 19.18) - open/close 7308 (time 1787148026.28 total 334.76 last 23.55) - open/close 7500 (time 1787148037.01 total 345.49 last 17.90) - open/close 7776 (time 1787148047.01 total 355.49 last 27.59) - open/close 7945 (time 1787148057.36 total 365.84 last 16.33) - open/close 8104 (time 1787148067.40 total 375.88 last 15.84) - open/close 8213 (time 1787148078.29 total 386.76 last 10.01) - open/close 8311 (time 1787148088.74 total 397.22 last 9.37) - open/close 8425 (time 1787148098.75 total 407.22 last 11.39) - open/close 8618 (time 1787148108.98 total 417.46 last 18.85) - open/close 8833 (time 1787148119.61 total 428.09 last 20.23) - open/close 8963 (time 1787148130.25 total 438.73 last 12.21) - open/close 9069 (time 1787148141.19 total 449.67 last 9.69) - open/close 9235 (time 1787148154.68 total 463.16 last 12.31) - open/close 9353 (time 1787148164.69 total 473.16 last 11.79) - open/close 9555 (time 1787148174.88 total 483.36 last 19.82) - open/close 9804 (time 1787148187.91 total 496.38 last 19.11) - open/close 10000 (time 1787148194.24 total 502.72 last 30.94) - open/close 10155 (time 1787148204.24 total 512.72 last 15.50) - open/close 10319 (time 1787148214.65 total 523.13 last 15.76) - open/close 10483 (time 1787148225.35 total 533.82 last 15.33) - open/close 10577 (time 1787148237.57 total 546.05 last 7.69) - open/close 10791 (time 1787148247.58 total 556.06 last 21.38) - open/close 10987 (time 1787148257.59 total 566.07 last 19.58) - open/close 11134 (time 1787148267.60 total 576.08 last 14.68) - open/close 11247 (time 1787148277.82 total 586.30 last 11.05) - open/close 11358 (time 1787148287.88 total 596.36 last 11.04) - open/close 11504 (time 1787148297.90 total 606.38 last 14.58) - open/close 11698 (time 1787148308.93 total 617.41 last 17.58) - open/close 11938 (time 1787148324.43 total 632.91 last 15.49) - open/close 12096 (time 1787148334.53 total 643.01 last 15.65) - open/close 12335 (time 1787148344.89 total 653.37 last 23.07) - open/close 12589 (time 1787148354.89 total 663.37 last 25.39) - open/close 12766 (time 1787148365.39 total 673.87 last 16.86) - open/close 12998 (time 1787148375.40 total 683.88 last 23.18) - open/close 13299 (time 1787148385.42 total 693.90 last 30.03) - open/close 13560 (time 1787148396.36 total 704.84 last 23.86) - open/close 13856 (time 1787148407.19 total 715.67 last 27.33) - open/close 14083 (time 1787148418.12 total 726.59 last 20.78) - open/close 14139 (time 1787148428.31 total 736.78 last 5.50) - open/close 14260 (time 1787148438.34 total 746.82 last 12.06) - open/close 14409 (time 1787148449.29 total 757.77 last 13.60) - open/close 14528 (time 1787148459.30 total 767.78 last 11.89) - open/close 14730 (time 1787148469.32 total 777.79 last 20.17) - open/close 14840 (time 1787148479.32 total 787.80 last 10.99) - open/close 15053 (time 1787148489.34 total 797.81 last 21.28) - open/close 15336 (time 1787148499.34 total 807.81 last 28.30) - open/close 15564 (time 1787148509.36 total 817.83 last 22.76) - open/close 15815 (time 1787148521.03 total 829.51 last 21.50) - open/close 15986 (time 1787148531.23 total 839.71 last 16.76) - open/close 16122 (time 1787148541.90 total 850.38 last 12.75) - open/close 16275 (time 1787148553.10 total 861.58 last 13.66) - open/close 16569 (time 1787148563.11 total 871.58 last 29.38) - open/close 16756 (time 1787148573.75 total 882.23 last 17.56) - open/close 17054 (time 1787148583.93 total 892.40 last 29.29) - open/close 17166 (time 1787148596.50 total 904.97 last 8.91) - open/close 17364 (time 1787148606.71 total 915.19 last 19.38) - open/close 17510 (time 1787148617.23 total 925.71 last 13.88) - open/close 17684 (time 1787148627.24 total 935.72 last 17.39) - open/close 17724 (time 1787148637.57 total 946.05 last 3.87) - open/close 17944 (time 1787148647.58 total 956.06 last 21.98) - open/close 18115 (time 1787148660.26 total 968.74 last 13.49) - open/close 18401 (time 1787148670.27 total 978.75 last 28.56) - open/close 18623 (time 1787148680.34 total 988.82 last 22.04) - open/close 18750 (time 1787148690.37 total 998.85 last 12.66) - open/close 18850 (time 1787148700.56 total 1009.04 last 9.82) - open/close 18994 (time 1787148711.45 total 1019.93 last 13.23) - open/close 19199 (time 1787148721.48 total 1029.96 last 20.43) - open/close 19291 (time 1787148731.49 total 1039.96 last 9.20) - open/close 20000 (time 1787148739.52 total 1048.00 last 88.23) - open/close 20912 (time 1787148749.53 total 1058.01 last 91.15) - open/close 21968 (time 1787148759.54 total 1068.01 last 105.50) - open/close 22958 (time 1787148769.56 total 1078.04 last 98.77) - open/close 23676 (time 1787148779.56 total 1088.04 last 71.77) - open/close 24451 (time 1787148789.57 total 1098.05 last 77.44) - open/close 25214 (time 1787148799.58 total 1108.05 last 76.26) - open/close 25994 (time 1787148809.59 total 1118.07 last 77.90) - open/close 26897 (time 1787148819.59 total 1128.07 last 90.28) - open/close 27563 (time 1787148829.59 total 1138.07 last 66.60) - open/close 28236 (time 1787148839.60 total 1148.07 last 67.28) - open/close 29204 (time 1787148849.60 total 1158.08 last 96.79) - open/close 30000 (time 1787148858.87 total 1167.35 last 85.80) - open/close 30909 (time 1787148868.88 total 1177.36 last 90.82) - open/close 31746 (time 1787148878.89 total 1187.37 last 83.63) - open/close 32786 (time 1787148888.89 total 1197.37 last 103.99) - open/close 33808 (time 1787148898.91 total 1207.38 last 102.08) - open/close 34838 (time 1787148908.91 total 1217.39 last 102.97) - open/close 35799 (time 1787148918.91 total 1227.39 last 96.06) - open/close 36773 (time 1787148928.92 total 1237.40 last 97.31) - open/close 37574 (time 1787148938.93 total 1247.41 last 80.05) - open/close 38453 (time 1787148948.94 total 1257.41 last 87.83) - open/close 39312 (time 1787148958.94 total 1267.42 last 85.88) - open/close 40000 (time 1787148966.12 total 1274.60 last 95.76) - open/close 41042 (time 1787148976.13 total 1284.60 last 104.18) - open/close 42049 (time 1787148986.13 total 1294.61 last 100.68) - open/close 42978 (time 1787148996.13 total 1304.61 last 92.89) - open/close 43831 (time 1787149006.13 total 1314.61 last 85.28) - open/close 44736 (time 1787149016.14 total 1324.61 last 90.46) - open/close 45547 (time 1787149026.14 total 1334.62 last 81.08) - open/close 46416 (time 1787149036.14 total 1344.62 last 86.88) - open/close 47414 (time 1787149046.15 total 1354.63 last 99.73) - open/close 48343 (time 1787149056.15 total 1364.63 last 92.85) - open/close 49188 (time 1787149066.16 total 1374.64 last 84.42) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1379.30 seconds: 35.98 ops/second - unlinked 0 (time 1787149075 ; total 0 ; last 0) - unlinked 10000 (time 1787149191 ; total 116 ; last 116) - unlinked 20000 (time 1787149292 ; total 217 ; last 101) - unlinked 30000 (time 1787149358 ; total 283 ; last 66) - unlinked 40000 (time 1787149420 ; total 345 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 403 seconds: 123.153847 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: modules unloaded. pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2