== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:20:23 (1781230823) start mds service on oleg345-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing load_modules_local oleg345-server: Loading modules from /home/green/git/lustre-release/lustre oleg345-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg345-server: detected 4 online CPUs by sysfs oleg345-server: MODOPTS_LIBCFS= oleg345-server: Force libcfs to create 2 CPU partitions oleg345-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg345-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg345-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg345-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg345-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock 192.168.203.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 192 (time 1781230926.81 total 10.62 last 18.08) - open/close 351 (time 1781230938.06 total 21.87 last 14.13) - open/close 578 (time 1781230949.03 total 32.84 last 20.70) - open/close 772 (time 1781230959.03 total 42.84 last 19.40) - open/close 1010 (time 1781230969.03 total 52.84 last 23.79) - open/close 1195 (time 1781230979.04 total 62.85 last 18.48) - open/close 1345 (time 1781230989.08 total 72.88 last 14.95) - open/close 1539 (time 1781230999.10 total 82.91 last 19.35) - open/close 1675 (time 1781231009.11 total 92.92 last 13.59) - open/close 1907 (time 1781231019.82 total 103.63 last 21.67) - open/close 2140 (time 1781231030.24 total 114.05 last 22.35) - open/close 2393 (time 1781231043.13 total 126.94 last 19.64) - open/close 2600 (time 1781231053.98 total 137.79 last 19.07) - open/close 2709 (time 1781231064.49 total 148.30 last 10.37) - open/close 2899 (time 1781231075.11 total 158.92 last 17.90) - open/close 3138 (time 1781231085.12 total 168.93 last 23.88) - open/close 3364 (time 1781231096.08 total 179.89 last 20.61) - open/close 3438 (time 1781231108.09 total 191.90 last 6.17) - open/close 3570 (time 1781231120.21 total 204.02 last 10.89) - open/close 3699 (time 1781231132.58 total 216.39 last 10.43) - open/close 3857 (time 1781231142.60 total 226.41 last 15.76) - open/close 3980 (time 1781231152.61 total 236.42 last 12.29) - open/close 4182 (time 1781231165.09 total 248.90 last 16.18) - open/close 4295 (time 1781231175.09 total 258.90 last 11.30) - open/close 4406 (time 1781231185.11 total 268.92 last 11.09) - open/close 4583 (time 1781231195.60 total 279.41 last 16.86) - open/close 4806 (time 1781231205.61 total 289.41 last 22.30) - open/close 4979 (time 1781231215.64 total 299.45 last 17.24) - open/close 5148 (time 1781231226.64 total 310.45 last 15.36) - open/close 5293 (time 1781231237.03 total 320.84 last 13.96) - open/close 5364 (time 1781231247.20 total 331.01 last 6.98) - open/close 5407 (time 1781231257.64 total 341.45 last 4.12) - open/close 5625 (time 1781231268.06 total 351.87 last 20.92) - open/close 5825 (time 1781231278.08 total 361.88 last 19.96) - open/close 6079 (time 1781231288.35 total 372.16 last 24.72) - open/close 6384 (time 1781231298.65 total 382.45 last 29.62) - open/close 6572 (time 1781231308.66 total 392.47 last 18.78) - open/close 6779 (time 1781231318.93 total 402.74 last 20.15) - open/close 6923 (time 1781231329.76 total 413.57 last 13.30) - open/close 7007 (time 1781231340.12 total 423.93 last 8.11) - open/close 7162 (time 1781231351.03 total 434.84 last 14.21) - open/close 7216 (time 1781231362.62 total 446.42 last 4.66) - open/close 7328 (time 1781231373.21 total 457.02 last 10.57) - open/close 7461 (time 1781231385.34 total 469.15 last 10.96) - open/close 7732 (time 1781231396.06 total 479.87 last 25.27) - open/close 7842 (time 1781231406.49 total 490.30 last 10.55) - open/close 8031 (time 1781231416.52 total 500.33 last 18.83) - open/close 8239 (time 1781231426.99 total 510.80 last 19.87) - open/close 8481 (time 1781231437.00 total 520.81 last 24.18) - open/close 8634 (time 1781231447.08 total 530.89 last 15.18) - open/close 8761 (time 1781231457.90 total 541.70 last 11.74) - open/close 8939 (time 1781231468.19 total 552.00 last 17.29) - open/close 9090 (time 1781231478.82 total 562.63 last 14.21) - open/close 9305 (time 1781231489.63 total 573.44 last 19.88) - open/close 9441 (time 1781231499.64 total 583.45 last 13.59) - open/close 9691 (time 1781231510.56 total 594.37 last 22.88) - open/close 9827 (time 1781231520.62 total 604.43 last 13.52) - open/close 10000 (time 1781231528.28 total 612.08 last 22.61) - open/close 10222 (time 1781231539.03 total 622.83 last 20.65) - open/close 10454 (time 1781231549.65 total 633.46 last 21.83) - open/close 10691 (time 1781231561.18 total 644.99 last 20.56) - open/close 10918 (time 1781231571.63 total 655.44 last 21.73) - open/close 11139 (time 1781231582.16 total 665.97 last 20.98) - open/close 11265 (time 1781231592.16 total 675.97 last 12.60) - open/close 11422 (time 1781231603.05 total 686.86 last 14.42) - open/close 11642 (time 1781231613.90 total 697.71 last 20.28) - open/close 11893 (time 1781231623.92 total 707.73 last 25.04) - open/close 12144 (time 1781231633.93 total 717.74 last 25.08) - open/close 12400 (time 1781231643.95 total 727.75 last 25.56) - open/close 12592 (time 1781231653.95 total 737.76 last 19.19) - open/close 12776 (time 1781231664.29 total 748.10 last 17.80) - open/close 13047 (time 1781231675.15 total 758.96 last 24.95) - open/close 13238 (time 1781231685.17 total 768.97 last 19.07) - open/close 13484 (time 1781231697.82 total 781.62 last 19.45) - open/close 13753 (time 1781231707.84 total 791.65 last 26.84) - open/close 13858 (time 1781231718.32 total 802.12 last 10.02) - open/close 14010 (time 1781231729.17 total 812.98 last 14.00) - open/close 14309 (time 1781231739.94 total 823.75 last 27.77) - open/close 14546 (time 1781231750.62 total 834.43 last 22.18) - open/close 14791 (time 1781231760.86 total 844.67 last 23.93) - open/close 15050 (time 1781231770.87 total 854.67 last 25.89) - open/close 15255 (time 1781231781.14 total 864.95 last 19.95) - open/close 15424 (time 1781231791.60 total 875.41 last 16.16) - open/close 15662 (time 1781231802.52 total 886.33 last 21.80) - open/close 15866 (time 1781231813.17 total 896.98 last 19.15) - open/close 16140 (time 1781231823.18 total 906.99 last 27.39) - open/close 16338 (time 1781231833.32 total 917.13 last 19.52) - open/close 16566 (time 1781231843.35 total 927.16 last 22.74) - open/close 16856 (time 1781231853.36 total 937.17 last 28.98) - open/close 17063 (time 1781231863.37 total 947.18 last 20.66) - open/close 17274 (time 1781231874.39 total 958.20 last 19.16) - open/close 17360 (time 1781231884.41 total 968.22 last 8.58) - open/close 17568 (time 1781231894.42 total 978.23 last 20.79) - open/close 17809 (time 1781231905.24 total 989.05 last 22.28) - open/close 18079 (time 1781231915.99 total 999.80 last 25.11) - open/close 18274 (time 1781231926.00 total 1009.81 last 19.48) - open/close 18447 (time 1781231936.01 total 1019.82 last 17.28) - open/close 19095 (time 1781231946.02 total 1029.83 last 64.76) - open/close 19737 (time 1781231956.03 total 1039.84 last 64.12) - open/close 20000 (time 1781231959.93 total 1043.74 last 67.39) - open/close 20751 (time 1781231969.95 total 1053.76 last 74.98) - open/close 21442 (time 1781231979.95 total 1063.76 last 69.07) - open/close 22174 (time 1781231989.96 total 1073.76 last 73.18) - open/close 22936 (time 1781231999.96 total 1083.77 last 76.16) - open/close 23620 (time 1781232009.97 total 1093.77 last 68.37) - open/close 24346 (time 1781232019.99 total 1103.80 last 72.45) - open/close 24896 (time 1781232030.00 total 1113.81 last 54.94) - open/close 25612 (time 1781232040.00 total 1123.81 last 71.57) - open/close 26267 (time 1781232050.00 total 1133.81 last 65.49) - open/close 27029 (time 1781232060.02 total 1143.83 last 76.04) - open/close 27807 (time 1781232070.07 total 1153.88 last 77.44) - open/close 28618 (time 1781232080.08 total 1163.89 last 81.03) - open/close 29323 (time 1781232090.09 total 1173.89 last 70.46) - open/close 30000 (time 1781232099.15 total 1182.96 last 74.70) - open/close 30681 (time 1781232109.16 total 1192.96 last 68.05) - open/close 31412 (time 1781232119.16 total 1202.97 last 73.08) - open/close 32085 (time 1781232129.17 total 1212.98 last 67.24) - open/close 32657 (time 1781232139.17 total 1222.98 last 57.16) - open/close 33134 (time 1781232149.19 total 1233.00 last 47.62) - open/close 33899 (time 1781232159.20 total 1243.01 last 76.43) - open/close 34588 (time 1781232169.20 total 1253.01 last 68.89) - open/close 35249 (time 1781232179.22 total 1263.03 last 65.96) - open/close 35960 (time 1781232189.22 total 1273.03 last 71.10) - open/close 36670 (time 1781232199.24 total 1283.05 last 70.90) - open/close 37375 (time 1781232209.25 total 1293.06 last 70.40) - open/close 38147 (time 1781232219.26 total 1303.07 last 77.17) - open/close 38858 (time 1781232229.26 total 1313.07 last 71.07) - open/close 39545 (time 1781232239.26 total 1323.07 last 68.67) - open/close 40000 (time 1781232246.23 total 1330.04 last 65.28) - open/close 40737 (time 1781232256.25 total 1340.06 last 73.58) - open/close 41406 (time 1781232266.26 total 1350.07 last 66.85) - open/close 42022 (time 1781232276.26 total 1360.07 last 61.59) - open/close 42584 (time 1781232286.27 total 1370.07 last 56.17) - open/close 43162 (time 1781232296.28 total 1380.09 last 57.71) - open/close 43831 (time 1781232306.30 total 1390.11 last 66.79) - open/close 44460 (time 1781232316.30 total 1400.11 last 62.90) - open/close 45120 (time 1781232326.31 total 1410.12 last 65.93) - open/close 45808 (time 1781232336.32 total 1420.13 last 68.74) - open/close 46502 (time 1781232346.32 total 1430.13 last 69.39) - open/close 47191 (time 1781232356.34 total 1440.15 last 68.75) - open/close 47852 (time 1781232366.34 total 1450.15 last 66.08) - open/close 48531 (time 1781232376.37 total 1460.18 last 67.70) - open/close 49209 (time 1781232386.38 total 1470.19 last 67.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1476.47 seconds: 33.61 ops/second - unlinked 0 (time 1781232397 ; total 0 ; last 0) - unlinked 10000 (time 1781232549 ; total 152 ; last 152) - unlinked 20000 (time 1781232695 ; total 298 ; last 146) - unlinked 30000 (time 1781232843 ; total 446 ; last 148) - unlinked 40000 (time 1781232995 ; total 598 ; last 152) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 736 seconds: 67.433426 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules via unload_modules_local on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local oleg345-server: modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2