== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:18:59 (1788445139) start mds service on oleg656-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg656-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 132 (time 1788445250.62 total 10.47 last 12.61) - open/close 257 (time 1788445260.63 total 20.48 last 12.49) - open/close 455 (time 1788445270.63 total 30.48 last 19.79) - open/close 726 (time 1788445281.19 total 41.04 last 25.67) - open/close 995 (time 1788445291.42 total 51.27 last 26.30) - open/close 1286 (time 1788445302.38 total 62.22 last 26.56) - open/close 1588 (time 1788445312.39 total 72.24 last 30.16) - open/close 1792 (time 1788445322.53 total 82.37 last 20.12) - open/close 1996 (time 1788445332.53 total 92.38 last 20.39) - open/close 2162 (time 1788445342.82 total 102.67 last 16.13) - open/close 2412 (time 1788445352.83 total 112.68 last 24.99) - open/close 2662 (time 1788445363.88 total 123.73 last 22.62) - open/close 2853 (time 1788445373.94 total 133.79 last 18.98) - open/close 3144 (time 1788445385.01 total 144.86 last 26.29) - open/close 3433 (time 1788445395.49 total 155.34 last 27.59) - open/close 3634 (time 1788445406.37 total 166.22 last 18.47) - open/close 3815 (time 1788445416.37 total 176.22 last 18.10) - open/close 3875 (time 1788445426.39 total 186.24 last 5.99) - open/close 4015 (time 1788445437.32 total 197.17 last 12.81) - open/close 4225 (time 1788445450.18 total 210.02 last 16.33) - open/close 4351 (time 1788445460.18 total 220.03 last 12.59) - open/close 4452 (time 1788445470.20 total 230.05 last 10.08) - open/close 4626 (time 1788445481.13 total 240.98 last 15.92) - open/close 4957 (time 1788445492.15 total 252.00 last 30.03) - open/close 5232 (time 1788445502.16 total 262.00 last 27.49) - open/close 5445 (time 1788445512.95 total 272.80 last 19.73) - open/close 5543 (time 1788445522.97 total 282.82 last 9.78) - open/close 5743 (time 1788445537.08 total 296.93 last 14.18) - open/close 5901 (time 1788445548.42 total 308.27 last 13.93) - open/close 6120 (time 1788445558.43 total 318.28 last 21.88) - open/close 6318 (time 1788445568.44 total 328.29 last 19.79) - open/close 6562 (time 1788445579.46 total 339.31 last 22.15) - open/close 6813 (time 1788445589.46 total 349.31 last 25.10) - open/close 6981 (time 1788445600.49 total 360.33 last 15.24) - open/close 7153 (time 1788445611.17 total 371.02 last 16.10) - open/close 7329 (time 1788445622.05 total 381.90 last 16.18) - open/close 7474 (time 1788445632.06 total 391.91 last 14.49) - open/close 7556 (time 1788445642.28 total 402.13 last 8.03) - open/close 7670 (time 1788445652.89 total 412.74 last 10.74) - open/close 7921 (time 1788445662.95 total 422.80 last 24.97) - open/close 7997 (time 1788445676.00 total 435.85 last 5.82) - open/close 8252 (time 1788445686.68 total 446.53 last 23.87) - open/close 8407 (time 1788445696.69 total 456.54 last 15.49) - open/close 8665 (time 1788445707.38 total 467.23 last 24.14) - open/close 8946 (time 1788445717.82 total 477.67 last 26.91) - open/close 9056 (time 1788445728.23 total 488.08 last 10.57) - open/close 9292 (time 1788445738.55 total 498.39 last 22.89) - open/close 9548 (time 1788445748.55 total 508.39 last 25.60) - open/close 9818 (time 1788445759.29 total 519.14 last 25.13) - open/close 10000 (time 1788445766.52 total 526.37 last 25.16) - open/close 10158 (time 1788445777.96 total 537.80 last 13.82) - open/close 10388 (time 1788445787.98 total 547.83 last 22.95) - open/close 10541 (time 1788445798.38 total 558.23 last 14.70) - open/close 10738 (time 1788445809.11 total 568.96 last 18.36) - open/close 11052 (time 1788445819.12 total 578.97 last 31.38) - open/close 11285 (time 1788445829.17 total 589.02 last 23.18) - open/close 11578 (time 1788445839.18 total 599.03 last 29.29) - open/close 11830 (time 1788445849.18 total 609.03 last 25.19) - open/close 12049 (time 1788445859.19 total 619.04 last 21.89) - open/close 12190 (time 1788445869.22 total 629.07 last 14.06) - open/close 12448 (time 1788445879.22 total 639.07 last 25.80) - open/close 12677 (time 1788445889.32 total 649.17 last 22.67) - open/close 12820 (time 1788445899.32 total 659.17 last 14.30) - open/close 12994 (time 1788445909.33 total 669.18 last 17.39) - open/close 13248 (time 1788445919.34 total 679.18 last 25.38) - open/close 13388 (time 1788445929.35 total 689.19 last 13.98) - open/close 13628 (time 1788445939.95 total 699.80 last 22.63) - open/close 13826 (time 1788445953.06 total 712.91 last 15.10) - open/close 13983 (time 1788445965.03 total 724.88 last 13.11) - open/close 14102 (time 1788445976.05 total 735.90 last 10.80) - open/close 14274 (time 1788445986.85 total 746.70 last 15.92) - open/close 14497 (time 1788445997.49 total 757.33 last 20.97) - open/close 14611 (time 1788446007.58 total 767.43 last 11.29) - open/close 14889 (time 1788446017.61 total 777.45 last 27.74) - open/close 15141 (time 1788446027.62 total 787.46 last 25.17) - open/close 15299 (time 1788446037.68 total 797.52 last 15.70) - open/close 15549 (time 1788446047.84 total 807.69 last 24.59) - open/close 15805 (time 1788446057.91 total 817.76 last 25.42) - open/close 16019 (time 1788446068.05 total 827.90 last 21.10) - open/close 16283 (time 1788446078.07 total 837.92 last 26.35) - open/close 16481 (time 1788446088.09 total 847.94 last 19.77) - open/close 16685 (time 1788446098.86 total 858.71 last 18.94) - open/close 16972 (time 1788446109.67 total 869.52 last 26.54) - open/close 17210 (time 1788446120.57 total 880.42 last 21.84) - open/close 17429 (time 1788446131.30 total 891.14 last 20.41) - open/close 17716 (time 1788446141.31 total 901.15 last 28.67) - open/close 18059 (time 1788446152.31 total 912.16 last 31.16) - open/close 18222 (time 1788446164.58 total 924.43 last 13.29) - open/close 18506 (time 1788446175.62 total 935.47 last 25.72) - open/close 18789 (time 1788446185.63 total 945.48 last 28.28) - open/close 19033 (time 1788446195.64 total 955.48 last 24.38) - open/close 19375 (time 1788446205.64 total 965.49 last 34.19) - open/close 19990 (time 1788446215.66 total 975.50 last 61.40) - open/close 20782 (time 1788446225.66 total 985.51 last 79.14) - open/close 21512 (time 1788446235.67 total 995.51 last 72.98) - open/close 22296 (time 1788446245.68 total 1005.53 last 78.31) - open/close 23102 (time 1788446255.69 total 1015.54 last 80.51) - open/close 23962 (time 1788446265.69 total 1025.54 last 85.99) - open/close 24807 (time 1788446275.69 total 1035.54 last 84.48) - open/close 25644 (time 1788446285.70 total 1045.55 last 83.63) - open/close 26379 (time 1788446295.72 total 1055.56 last 73.39) - open/close 27148 (time 1788446305.72 total 1065.56 last 76.89) - open/close 27956 (time 1788446315.72 total 1075.57 last 80.76) - open/close 28763 (time 1788446325.73 total 1085.57 last 80.66) - open/close 29615 (time 1788446335.73 total 1095.58 last 85.14) - open/close 30000 (time 1788446340.53 total 1100.38 last 80.25) - open/close 30800 (time 1788446350.55 total 1110.40 last 79.87) - open/close 31637 (time 1788446360.55 total 1120.40 last 83.64) - open/close 32448 (time 1788446370.57 total 1130.41 last 81.01) - open/close 33268 (time 1788446380.57 total 1140.42 last 81.97) - open/close 34106 (time 1788446390.57 total 1150.42 last 83.79) - open/close 34833 (time 1788446400.58 total 1160.43 last 72.61) - open/close 35607 (time 1788446410.60 total 1170.44 last 77.29) - open/close 36380 (time 1788446420.61 total 1180.46 last 77.18) - open/close 37188 (time 1788446430.62 total 1190.47 last 80.71) - open/close 38043 (time 1788446440.64 total 1200.49 last 85.37) - open/close 38820 (time 1788446450.66 total 1210.50 last 77.56) - open/close 39602 (time 1788446460.66 total 1220.51 last 78.17) - open/close 40000 (time 1788446465.76 total 1225.61 last 77.98) - open/close 40761 (time 1788446475.78 total 1235.62 last 76.01) - open/close 41547 (time 1788446485.78 total 1245.63 last 78.53) - open/close 42334 (time 1788446495.79 total 1255.64 last 78.63) - open/close 43145 (time 1788446505.81 total 1265.66 last 80.96) - open/close 43919 (time 1788446515.81 total 1275.66 last 77.37) - open/close 44752 (time 1788446525.82 total 1285.67 last 83.26) - open/close 45586 (time 1788446535.82 total 1295.67 last 83.40) - open/close 46359 (time 1788446545.84 total 1305.69 last 77.15) - open/close 47199 (time 1788446555.84 total 1315.69 last 84.00) - open/close 48002 (time 1788446565.85 total 1325.70 last 80.19) - open/close 48758 (time 1788446575.85 total 1335.70 last 75.60) - open/close 49540 (time 1788446585.86 total 1345.71 last 78.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1346.94 seconds: 36.85 ops/second - unlinked 0 (time 1788446591 ; total 0 ; last 0) - unlinked 10000 (time 1788446726 ; total 135 ; last 135) - unlinked 20000 (time 1788446861 ; total 270 ; last 135) - unlinked 30000 (time 1788446995 ; total 404 ; last 134) - unlinked 40000 (time 1788447132 ; total 541 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 670 seconds: 74.076118 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: modules unloaded. pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 pdsh@oleg656-client: oleg656-client: ssh exited with exit code 2