== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:52:42 (1787392362) start mds service on oleg624-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg624-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg624-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg624-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg624-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1787392467.45 total 10.33 last 18.21) - open/close 400 (time 1787392477.46 total 20.33 last 21.19) - open/close 601 (time 1787392487.47 total 30.35 last 20.07) - open/close 865 (time 1787392497.48 total 40.35 last 26.40) - open/close 1140 (time 1787392507.48 total 50.35 last 27.49) - open/close 1352 (time 1787392517.48 total 60.35 last 21.20) - open/close 1577 (time 1787392527.60 total 70.47 last 22.24) - open/close 1793 (time 1787392537.61 total 80.49 last 21.57) - open/close 2011 (time 1787392547.63 total 90.51 last 21.76) - open/close 2128 (time 1787392560.18 total 103.05 last 9.33) - open/close 2410 (time 1787392570.19 total 113.07 last 28.15) - open/close 2671 (time 1787392580.22 total 123.09 last 26.04) - open/close 2947 (time 1787392590.22 total 133.09 last 27.60) - open/close 3214 (time 1787392600.23 total 143.11 last 26.66) - open/close 3397 (time 1787392610.53 total 153.40 last 17.77) - open/close 3631 (time 1787392620.54 total 163.41 last 23.39) - open/close 3877 (time 1787392630.56 total 173.43 last 24.55) - open/close 4175 (time 1787392640.59 total 183.46 last 29.71) - open/close 4371 (time 1787392651.31 total 194.18 last 18.28) - open/close 4623 (time 1787392662.13 total 205.00 last 23.30) - open/close 4886 (time 1787392672.24 total 215.11 last 26.02) - open/close 5045 (time 1787392683.08 total 225.95 last 14.67) - open/close 5286 (time 1787392693.18 total 236.05 last 23.86) - open/close 5528 (time 1787392703.63 total 246.50 last 23.17) - open/close 5792 (time 1787392713.64 total 256.52 last 26.36) - open/close 6022 (time 1787392724.17 total 267.04 last 21.85) - open/close 6280 (time 1787392735.03 total 277.90 last 23.77) - open/close 6508 (time 1787392745.03 total 287.90 last 22.80) - open/close 6724 (time 1787392755.51 total 298.38 last 20.60) - open/close 6982 (time 1787392765.93 total 308.80 last 24.77) - open/close 7230 (time 1787392775.93 total 318.80 last 24.79) - open/close 7474 (time 1787392787.00 total 329.87 last 22.04) - open/close 7752 (time 1787392798.06 total 340.93 last 25.14) - open/close 8058 (time 1787392809.06 total 351.94 last 27.81) - open/close 8298 (time 1787392819.44 total 362.31 last 23.13) - open/close 8620 (time 1787392829.45 total 372.32 last 32.17) - open/close 8864 (time 1787392839.70 total 382.57 last 23.79) - open/close 9160 (time 1787392849.71 total 392.58 last 29.57) - open/close 9397 (time 1787392860.41 total 403.28 last 22.15) - open/close 9565 (time 1787392874.81 total 417.69 last 11.66) - open/close 9718 (time 1787392885.55 total 428.42 last 14.25) - open/close 9891 (time 1787392895.61 total 438.48 last 17.20) - open/close 10000 (time 1787392898.98 total 441.85 last 32.34) - open/close 10207 (time 1787392908.99 total 451.87 last 20.67) - open/close 10367 (time 1787392918.99 total 461.87 last 16.00) - open/close 10690 (time 1787392929.84 total 472.72 last 29.77) - open/close 10899 (time 1787392940.91 total 483.78 last 18.89) - open/close 11012 (time 1787392953.45 total 496.32 last 9.01) - open/close 11264 (time 1787392964.33 total 507.20 last 23.17) - open/close 11563 (time 1787392974.67 total 517.54 last 28.92) - open/close 11806 (time 1787392985.01 total 527.88 last 23.50) - open/close 12103 (time 1787392995.33 total 538.20 last 28.78) - open/close 12374 (time 1787393005.33 total 548.21 last 27.08) - open/close 12606 (time 1787393015.34 total 558.21 last 23.18) - open/close 12852 (time 1787393025.35 total 568.22 last 24.59) - open/close 13094 (time 1787393035.37 total 578.24 last 24.14) - open/close 13299 (time 1787393045.38 total 588.26 last 20.47) - open/close 13435 (time 1787393057.26 total 600.13 last 11.45) - open/close 13578 (time 1787393067.28 total 610.15 last 14.27) - open/close 13837 (time 1787393077.28 total 620.16 last 25.90) - open/close 14053 (time 1787393087.29 total 630.16 last 21.59) - open/close 14300 (time 1787393097.90 total 640.77 last 23.28) - open/close 14558 (time 1787393108.56 total 651.43 last 24.20) - open/close 14681 (time 1787393119.23 total 662.11 last 11.53) - open/close 14813 (time 1787393129.23 total 672.11 last 13.20) - open/close 15064 (time 1787393140.04 total 682.91 last 23.23) - open/close 15216 (time 1787393150.42 total 693.30 last 14.64) - open/close 15429 (time 1787393160.88 total 703.75 last 20.38) - open/close 15612 (time 1787393171.89 total 714.77 last 16.61) - open/close 15802 (time 1787393182.65 total 725.53 last 17.66) - open/close 15985 (time 1787393193.17 total 736.05 last 17.40) - open/close 16142 (time 1787393208.25 total 751.13 last 10.41) - open/close 16398 (time 1787393218.94 total 761.82 last 23.95) - open/close 16664 (time 1787393229.58 total 772.45 last 25.01) - open/close 16928 (time 1787393239.59 total 782.46 last 26.37) - open/close 17212 (time 1787393249.66 total 792.53 last 28.20) - open/close 17459 (time 1787393260.60 total 803.47 last 22.57) - open/close 17596 (time 1787393270.61 total 813.48 last 13.69) - open/close 17798 (time 1787393281.19 total 824.07 last 19.08) - open/close 18017 (time 1787393292.58 total 835.46 last 19.23) - open/close 18230 (time 1787393302.83 total 845.71 last 20.78) - open/close 18524 (time 1787393312.85 total 855.72 last 29.36) - open/close 18664 (time 1787393322.90 total 865.77 last 13.93) - open/close 18844 (time 1787393333.46 total 876.33 last 17.05) - open/close 18979 (time 1787393343.48 total 886.35 last 13.47) - open/close 19217 (time 1787393353.51 total 896.38 last 23.73) - open/close 19479 (time 1787393363.53 total 906.40 last 26.14) - open/close 19602 (time 1787393373.54 total 916.41 last 12.28) - open/close 19682 (time 1787393383.57 total 926.44 last 7.98) - open/close 19759 (time 1787393394.62 total 937.49 last 6.97) - open/close 19848 (time 1787393404.63 total 947.50 last 8.89) - open/close 20000 (time 1787393413.16 total 956.03 last 17.83) - open/close 20566 (time 1787393423.16 total 966.03 last 56.58) - open/close 21310 (time 1787393433.17 total 976.04 last 74.34) - open/close 22013 (time 1787393443.19 total 986.07 last 70.14) - open/close 22583 (time 1787393453.19 total 996.07 last 56.99) - open/close 23375 (time 1787393463.20 total 1006.08 last 79.12) - open/close 24122 (time 1787393473.21 total 1016.08 last 74.64) - open/close 24842 (time 1787393483.21 total 1026.09 last 71.98) - open/close 25662 (time 1787393493.22 total 1036.09 last 81.98) - open/close 26459 (time 1787393503.23 total 1046.11 last 79.56) - open/close 27134 (time 1787393513.24 total 1056.11 last 67.48) - open/close 27913 (time 1787393523.25 total 1066.12 last 77.81) - open/close 28715 (time 1787393533.25 total 1076.12 last 80.18) - open/close 29523 (time 1787393543.25 total 1086.12 last 80.80) - open/close 30000 (time 1787393549.70 total 1092.57 last 74.03) - open/close 30746 (time 1787393559.70 total 1102.57 last 74.59) - open/close 31430 (time 1787393569.71 total 1112.58 last 68.31) - open/close 32172 (time 1787393579.71 total 1122.58 last 74.20) - open/close 32940 (time 1787393589.72 total 1132.59 last 76.73) - open/close 33637 (time 1787393599.73 total 1142.60 last 69.65) - open/close 34370 (time 1787393609.73 total 1152.60 last 73.28) - open/close 35059 (time 1787393619.74 total 1162.62 last 68.80) - open/close 35792 (time 1787393629.75 total 1172.62 last 73.26) - open/close 36514 (time 1787393639.76 total 1182.63 last 72.14) - open/close 37174 (time 1787393649.76 total 1192.63 last 66.00) - open/close 37713 (time 1787393659.76 total 1202.63 last 53.89) - open/close 38416 (time 1787393669.77 total 1212.64 last 70.22) - open/close 39127 (time 1787393679.77 total 1222.64 last 71.10) - open/close 39848 (time 1787393689.78 total 1232.65 last 72.07) - open/close 40000 (time 1787393692.19 total 1235.06 last 62.91) - open/close 40669 (time 1787393702.20 total 1245.07 last 66.86) - open/close 41291 (time 1787393712.20 total 1255.07 last 62.17) - open/close 41879 (time 1787393722.22 total 1265.09 last 58.70) - open/close 42550 (time 1787393732.22 total 1275.10 last 67.08) - open/close 43341 (time 1787393742.23 total 1285.10 last 79.04) - open/close 44107 (time 1787393752.23 total 1295.11 last 76.57) - open/close 44761 (time 1787393762.24 total 1305.11 last 65.36) - open/close 45402 (time 1787393772.25 total 1315.12 last 64.03) - open/close 46125 (time 1787393782.27 total 1325.15 last 72.13) - open/close 46878 (time 1787393792.28 total 1335.15 last 75.23) - open/close 47567 (time 1787393802.28 total 1345.15 last 68.90) - open/close 48209 (time 1787393812.29 total 1355.16 last 64.18) - open/close 48937 (time 1787393822.29 total 1365.17 last 72.75) - open/close 49601 (time 1787393832.30 total 1375.17 last 66.35) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1375.69 seconds: 36.08 ops/second - unlinked 0 (time 1787393837 ; total 0 ; last 0) - unlinked 10000 (time 1787393978 ; total 141 ; last 141) - unlinked 20000 (time 1787394119 ; total 282 ; last 141) - unlinked 30000 (time 1787394262 ; total 425 ; last 143) - unlinked 40000 (time 1787394403 ; total 566 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 694 seconds: 71.514412 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg624-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg624-server unloading modules via unload_modules_local on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing unload_modules_local oleg624-server: modules unloaded. pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2