== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:23:10 (1786191790) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1786191879.37 total 10.09 last 20.71) - open/close 422 (time 1786191889.38 total 20.10 last 21.28) - open/close 649 (time 1786191900.11 total 30.84 last 21.15) - open/close 835 (time 1786191910.28 total 41.01 last 18.29) - open/close 1012 (time 1786191920.79 total 51.52 last 16.84) - open/close 1161 (time 1786191930.79 total 61.52 last 14.89) - open/close 1341 (time 1786191941.51 total 72.23 last 16.80) - open/close 1508 (time 1786191951.88 total 82.60 last 16.10) - open/close 1653 (time 1786191961.93 total 92.65 last 14.43) - open/close 1775 (time 1786191972.81 total 103.53 last 11.21) - open/close 1911 (time 1786191982.92 total 113.65 last 13.44) - open/close 2116 (time 1786191993.74 total 124.47 last 18.95) - open/close 2311 (time 1786192004.42 total 135.15 last 18.26) - open/close 2435 (time 1786192015.30 total 146.03 last 11.40) - open/close 2579 (time 1786192025.44 total 156.17 last 14.20) - open/close 2680 (time 1786192036.17 total 166.90 last 9.41) - open/close 2802 (time 1786192046.92 total 177.65 last 11.35) - open/close 2943 (time 1786192057.87 total 188.60 last 12.88) - open/close 3063 (time 1786192068.43 total 199.15 last 11.37) - open/close 3174 (time 1786192078.98 total 209.71 last 10.52) - open/close 3253 (time 1786192089.23 total 219.96 last 7.71) - open/close 3435 (time 1786192100.04 total 230.77 last 16.84) - open/close 3572 (time 1786192110.05 total 240.78 last 13.69) - open/close 3664 (time 1786192120.60 total 251.32 last 8.72) - open/close 3837 (time 1786192130.96 total 261.68 last 16.70) - open/close 4044 (time 1786192141.71 total 272.43 last 19.26) - open/close 4154 (time 1786192152.14 total 282.87 last 10.54) - open/close 4192 (time 1786192162.95 total 293.68 last 3.52) - open/close 4301 (time 1786192173.39 total 304.12 last 10.44) - open/close 4416 (time 1786192184.01 total 314.73 last 10.84) - open/close 4644 (time 1786192194.38 total 325.11 last 21.97) - open/close 4996 (time 1786192204.68 total 335.41 last 34.18) - open/close 5319 (time 1786192214.69 total 345.41 last 32.28) - open/close 5554 (time 1786192225.49 total 356.22 last 21.74) - open/close 5724 (time 1786192235.50 total 366.23 last 16.99) - open/close 6006 (time 1786192246.22 total 376.94 last 26.31) - open/close 6132 (time 1786192256.91 total 387.64 last 11.78) - open/close 6301 (time 1786192266.91 total 397.64 last 16.90) - open/close 6366 (time 1786192277.06 total 407.79 last 6.40) - open/close 6472 (time 1786192287.43 total 418.15 last 10.23) - open/close 6546 (time 1786192299.60 total 430.32 last 6.08) - open/close 6702 (time 1786192310.48 total 441.21 last 14.33) - open/close 6888 (time 1786192320.49 total 451.21 last 18.59) - open/close 7110 (time 1786192330.88 total 461.61 last 21.35) - open/close 7179 (time 1786192340.89 total 471.62 last 6.89) - open/close 7268 (time 1786192352.33 total 483.05 last 7.78) - open/close 7528 (time 1786192363.01 total 493.74 last 24.33) - open/close 7890 (time 1786192373.26 total 503.98 last 35.33) - open/close 8174 (time 1786192384.27 total 514.99 last 25.79) - open/close 8261 (time 1786192394.37 total 525.10 last 8.61) - open/close 8521 (time 1786192404.38 total 535.11 last 25.97) - open/close 8693 (time 1786192414.39 total 545.12 last 17.19) - open/close 8835 (time 1786192425.03 total 555.75 last 13.35) - open/close 9142 (time 1786192435.03 total 565.76 last 30.69) - open/close 9578 (time 1786192445.03 total 575.76 last 43.59) - open/close 10000 (time 1786192448.52 total 579.24 last 121.11) - open/close 11296 (time 1786192458.53 total 589.25 last 129.51) - open/close 12534 (time 1786192468.53 total 599.26 last 123.71) - open/close 13828 (time 1786192478.53 total 609.26 last 129.39) - open/close 15284 (time 1786192488.54 total 619.26 last 145.56) - open/close 16610 (time 1786192498.54 total 629.27 last 132.55) - open/close 17967 (time 1786192508.55 total 639.27 last 135.61) - open/close 19518 (time 1786192518.55 total 649.28 last 155.01) - open/close 20000 (time 1786192521.42 total 652.14 last 168.23) - open/close 21404 (time 1786192531.42 total 662.15 last 140.32) - open/close 22742 (time 1786192541.43 total 672.15 last 133.76) - open/close 24032 (time 1786192551.43 total 682.15 last 128.98) - open/close 25387 (time 1786192561.43 total 692.16 last 135.42) - open/close 26707 (time 1786192571.44 total 702.16 last 131.95) - open/close 28068 (time 1786192581.44 total 712.17 last 136.07) - open/close 29311 (time 1786192591.44 total 722.17 last 124.26) - open/close 30000 (time 1786192597.64 total 728.36 last 111.24) - open/close 31053 (time 1786192607.65 total 738.37 last 105.21) - open/close 32280 (time 1786192617.65 total 748.38 last 122.64) - open/close 33417 (time 1786192627.65 total 758.38 last 113.64) - open/close 34497 (time 1786192637.66 total 768.39 last 107.94) - open/close 35664 (time 1786192647.66 total 778.39 last 116.68) - open/close 36704 (time 1786192657.67 total 788.40 last 103.93) - open/close 37803 (time 1786192667.67 total 798.40 last 109.89) - open/close 39005 (time 1786192677.68 total 808.41 last 120.08) - open/close 40000 (time 1786192687.07 total 817.80 last 105.95) - open/close 41322 (time 1786192697.07 total 827.80 last 132.20) - open/close 42559 (time 1786192707.07 total 837.80 last 123.67) - open/close 43663 (time 1786192717.08 total 847.81 last 110.33) - open/close 44681 (time 1786192727.09 total 857.81 last 101.74) - open/close 45744 (time 1786192737.09 total 867.81 last 106.30) - open/close 46751 (time 1786192747.09 total 877.81 last 100.69) - open/close 47879 (time 1786192757.09 total 887.82 last 112.76) - open/close 49068 (time 1786192767.10 total 897.82 last 118.82) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 902.83 seconds: 54.97 ops/second - unlinked 0 (time 1786192774 ; total 0 ; last 0) - unlinked 10000 (time 1786192838 ; total 64 ; last 64) - unlinked 20000 (time 1786192871 ; total 97 ; last 33) - unlinked 30000 (time 1786192901 ; total 127 ; last 30) - unlinked 40000 (time 1786192932 ; total 158 ; last 31) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 189 seconds: 262.597870 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2