== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:09:37 (1789132177) start mds service on oleg143-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg143-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg143-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1789132315.36 total 16.05 last 8.85) - open/close 243 (time 1789132327.95 total 28.64 last 8.02) - open/close 427 (time 1789132338.98 total 39.67 last 16.68) - open/close 507 (time 1789132349.07 total 49.76 last 7.93) - open/close 679 (time 1789132359.07 total 59.76 last 17.20) - open/close 748 (time 1789132369.51 total 70.21 last 6.61) - open/close 809 (time 1789132379.56 total 80.25 last 6.07) - open/close 968 (time 1789132389.71 total 90.40 last 15.66) - open/close 1101 (time 1789132400.67 total 101.36 last 12.13) - open/close 1321 (time 1789132410.69 total 111.39 last 21.95) - open/close 1466 (time 1789132420.70 total 121.39 last 14.49) - open/close 1663 (time 1789132430.71 total 131.41 last 19.68) - open/close 1721 (time 1789132441.38 total 142.07 last 5.44) - open/close 1765 (time 1789132451.67 total 152.37 last 4.27) - open/close 2033 (time 1789132461.69 total 162.38 last 26.75) - open/close 2151 (time 1789132472.37 total 173.06 last 11.05) - open/close 2411 (time 1789132482.38 total 183.07 last 25.98) - open/close 2659 (time 1789132492.40 total 193.09 last 24.75) - open/close 2787 (time 1789132505.28 total 205.97 last 9.94) - open/close 3039 (time 1789132516.13 total 216.82 last 23.22) - open/close 3197 (time 1789132526.13 total 226.82 last 15.79) - open/close 3329 (time 1789132536.18 total 236.87 last 13.13) - open/close 3563 (time 1789132546.65 total 247.34 last 22.37) - open/close 3797 (time 1789132557.27 total 257.96 last 22.03) - open/close 4099 (time 1789132567.27 total 267.96 last 30.20) - open/close 4289 (time 1789132577.28 total 277.97 last 18.99) - open/close 4547 (time 1789132587.60 total 288.29 last 24.98) - open/close 4754 (time 1789132599.85 total 300.54 last 16.90) - open/close 4972 (time 1789132609.86 total 310.55 last 21.78) - open/close 5266 (time 1789132619.88 total 320.57 last 29.33) - open/close 5465 (time 1789132630.50 total 331.19 last 18.74) - open/close 5671 (time 1789132640.66 total 341.35 last 20.27) - open/close 5894 (time 1789132650.67 total 351.36 last 22.29) - open/close 6063 (time 1789132660.67 total 361.36 last 16.90) - open/close 6210 (time 1789132670.71 total 371.40 last 14.64) - open/close 6358 (time 1789132681.56 total 382.25 last 13.64) - open/close 6440 (time 1789132691.75 total 392.44 last 8.05) - open/close 6672 (time 1789132701.75 total 402.44 last 23.19) - open/close 6717 (time 1789132712.20 total 412.89 last 4.31) - open/close 6881 (time 1789132722.84 total 423.53 last 15.41) - open/close 6978 (time 1789132733.84 total 434.53 last 8.82) - open/close 7141 (time 1789132744.54 total 445.24 last 15.22) - open/close 7273 (time 1789132754.56 total 455.25 last 13.18) - open/close 7411 (time 1789132764.63 total 465.32 last 13.70) - open/close 7541 (time 1789132774.64 total 475.33 last 12.99) - open/close 7729 (time 1789132785.37 total 486.06 last 17.53) - open/close 7985 (time 1789132796.18 total 496.87 last 23.67) - open/close 8212 (time 1789132807.20 total 507.89 last 20.60) - open/close 8344 (time 1789132820.00 total 520.69 last 10.32) - open/close 8490 (time 1789132830.00 total 530.69 last 14.59) - open/close 8681 (time 1789132842.21 total 542.90 last 15.64) - open/close 8867 (time 1789132854.57 total 555.26 last 15.06) - open/close 8995 (time 1789132864.81 total 565.50 last 12.50) - open/close 9099 (time 1789132874.96 total 575.65 last 10.24) - open/close 9283 (time 1789132888.03 total 588.72 last 14.08) - open/close 9419 (time 1789132898.36 total 599.05 last 13.17) - open/close 9551 (time 1789132908.36 total 609.05 last 13.20) - open/close 9759 (time 1789132918.67 total 619.37 last 20.16) - open/close 9975 (time 1789132930.33 total 631.02 last 18.53) - open/close 10000 (time 1789132932.92 total 633.61 last 9.64) - open/close 10153 (time 1789132942.93 total 643.62 last 15.29) - open/close 10323 (time 1789132953.07 total 653.76 last 16.75) - open/close 10440 (time 1789132963.62 total 664.31 last 11.09) - open/close 10598 (time 1789132974.02 total 674.71 last 15.19) - open/close 10786 (time 1789132984.03 total 684.72 last 18.79) - open/close 10962 (time 1789132994.04 total 694.73 last 17.57) - open/close 11105 (time 1789133004.70 total 705.40 last 13.41) - open/close 11224 (time 1789133015.59 total 716.28 last 10.93) - open/close 11465 (time 1789133026.15 total 726.84 last 22.82) - open/close 11746 (time 1789133036.16 total 736.86 last 28.06) - open/close 11999 (time 1789133046.18 total 746.87 last 25.26) - open/close 12268 (time 1789133056.20 total 756.90 last 26.84) - open/close 12319 (time 1789133067.26 total 767.95 last 4.61) - open/close 12431 (time 1789133077.27 total 777.96 last 11.19) - open/close 12551 (time 1789133087.27 total 787.97 last 12.00) - open/close 12643 (time 1789133097.70 total 798.39 last 8.82) - open/close 12682 (time 1789133108.19 total 808.88 last 3.72) - open/close 12824 (time 1789133118.24 total 818.93 last 14.14) - open/close 12973 (time 1789133128.24 total 828.93 last 14.89) - open/close 13104 (time 1789133139.16 total 839.85 last 12.00) - open/close 13217 (time 1789133149.33 total 850.02 last 11.11) - open/close 13423 (time 1789133159.40 total 860.09 last 20.46) - open/close 13592 (time 1789133169.41 total 870.10 last 16.88) - open/close 13698 (time 1789133179.55 total 880.24 last 10.46) - open/close 13907 (time 1789133190.23 total 890.92 last 19.56) - open/close 14128 (time 1789133200.29 total 900.98 last 21.98) - open/close 14289 (time 1789133210.36 total 911.05 last 15.99) - open/close 14449 (time 1789133223.46 total 924.15 last 12.21) - open/close 14660 (time 1789133233.47 total 934.17 last 21.07) - open/close 14761 (time 1789133243.95 total 944.64 last 9.64) - open/close 14903 (time 1789133255.21 total 955.90 last 12.61) - open/close 15033 (time 1789133265.89 total 966.58 last 12.18) - open/close 15206 (time 1789133276.68 total 977.37 last 16.04) - open/close 15321 (time 1789133287.71 total 988.40 last 10.42) - open/close 15546 (time 1789133300.18 total 1000.87 last 18.04) - open/close 15784 (time 1789133310.20 total 1010.89 last 23.77) - open/close 16074 (time 1789133320.20 total 1020.89 last 28.99) - open/close 16353 (time 1789133330.21 total 1030.90 last 27.88) - open/close 16608 (time 1789133340.69 total 1041.38 last 24.33) - open/close 16866 (time 1789133350.70 total 1051.39 last 25.77) - open/close 17138 (time 1789133361.61 total 1062.30 last 24.92) - open/close 17322 (time 1789133371.93 total 1072.62 last 17.83) - open/close 18186 (time 1789133381.94 total 1082.63 last 86.32) - open/close 19044 (time 1789133391.95 total 1092.64 last 85.74) - open/close 19937 (time 1789133401.95 total 1102.65 last 89.24) - open/close 20939 (time 1789133411.95 total 1112.65 last 100.20) - open/close 21816 (time 1789133421.96 total 1122.65 last 87.66) - open/close 22682 (time 1789133431.97 total 1132.66 last 86.49) - open/close 23402 (time 1789133441.98 total 1142.67 last 71.93) - open/close 24200 (time 1789133451.99 total 1152.68 last 79.70) - open/close 25126 (time 1789133462.01 total 1162.70 last 92.48) - open/close 25746 (time 1789133472.02 total 1172.71 last 61.93) - open/close 26543 (time 1789133482.02 total 1182.71 last 79.68) - open/close 27410 (time 1789133492.04 total 1192.73 last 86.54) - open/close 28405 (time 1789133502.05 total 1202.74 last 99.42) - open/close 29293 (time 1789133512.05 total 1212.74 last 88.75) - open/close 30000 (time 1789133519.35 total 1220.04 last 96.93) - open/close 31062 (time 1789133529.35 total 1230.05 last 106.10) - open/close 31982 (time 1789133539.36 total 1240.05 last 91.95) - open/close 32852 (time 1789133549.37 total 1250.06 last 86.93) - open/close 33673 (time 1789133559.37 total 1260.07 last 82.05) - open/close 34394 (time 1789133569.38 total 1270.07 last 72.10) - open/close 35157 (time 1789133579.38 total 1280.07 last 76.27) - open/close 35634 (time 1789133589.39 total 1290.08 last 47.63) - open/close 35898 (time 1789133599.40 total 1300.09 last 26.39) - open/close 36644 (time 1789133609.40 total 1310.09 last 74.57) - open/close 37118 (time 1789133619.40 total 1320.09 last 47.38) - open/close 37587 (time 1789133629.41 total 1330.10 last 46.87) - open/close 38231 (time 1789133639.42 total 1340.11 last 64.35) - open/close 38740 (time 1789133649.44 total 1350.13 last 50.80) - open/close 39199 (time 1789133659.44 total 1360.13 last 45.89) - open/close 39895 (time 1789133669.45 total 1370.14 last 69.54) - open/close 40924 (time 1789133679.45 total 1380.14 last 102.86) - open/close 41751 (time 1789133689.47 total 1390.16 last 82.57) - open/close 42502 (time 1789133699.47 total 1400.17 last 75.05) - open/close 43390 (time 1789133709.48 total 1410.17 last 88.73) - open/close 43954 (time 1789133719.49 total 1420.18 last 56.38) - open/close 44771 (time 1789133729.49 total 1430.19 last 81.64) - open/close 45623 (time 1789133739.50 total 1440.19 last 85.14) - open/close 46346 (time 1789133749.51 total 1450.20 last 72.23) - open/close 47202 (time 1789133759.52 total 1460.21 last 85.53) - open/close 48078 (time 1789133769.53 total 1470.22 last 87.52) - open/close 48916 (time 1789133779.54 total 1480.23 last 83.72) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No space left on device total: 49630 open/close in 1489.59 seconds: 33.32 ops/second - unlinked 0 (time 1789133793 ; total 0 ; last 0) - unlinked 10000 (time 1789133922 ; total 129 ; last 129) - unlinked 20000 (time 1789134057 ; total 264 ; last 135) - unlinked 30000 (time 1789134190 ; total 397 ; last 133) - unlinked 40000 (time 1789134310 ; total 517 ; last 120) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49630) error: No such file or directory total: 49630 unlinks in 650 seconds: 76.353844 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg143-server unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2