== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:32:01 (1788906721) start mds service on oleg343-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1788906816.55 total 10.18 last 21.22) - open/close 473 (time 1788906826.55 total 20.18 last 25.69) - open/close 657 (time 1788906836.71 total 30.34 last 18.11) - open/close 909 (time 1788906847.12 total 40.75 last 24.20) - open/close 991 (time 1788906858.39 total 52.02 last 7.28) - open/close 1278 (time 1788906868.39 total 62.02 last 28.70) - open/close 1454 (time 1788906878.40 total 72.02 last 17.59) - open/close 1718 (time 1788906889.11 total 82.74 last 24.64) - open/close 2057 (time 1788906899.29 total 92.92 last 33.31) - open/close 2390 (time 1788906909.91 total 103.54 last 31.33) - open/close 2692 (time 1788906920.53 total 114.16 last 28.44) - open/close 3021 (time 1788906931.35 total 124.98 last 30.41) - open/close 3356 (time 1788906942.37 total 136.00 last 30.42) - open/close 3630 (time 1788906953.05 total 146.68 last 25.65) - open/close 3933 (time 1788906963.05 total 156.68 last 30.28) - open/close 4139 (time 1788906973.27 total 166.90 last 20.16) - open/close 4315 (time 1788906983.29 total 176.91 last 17.57) - open/close 4649 (time 1788906993.30 total 186.93 last 33.36) - open/close 4874 (time 1788907004.12 total 197.75 last 20.79) - open/close 5202 (time 1788907014.87 total 208.50 last 30.51) - open/close 5555 (time 1788907025.82 total 219.45 last 32.24) - open/close 5830 (time 1788907036.19 total 229.82 last 26.53) - open/close 6087 (time 1788907046.29 total 239.92 last 25.44) - open/close 6353 (time 1788907056.67 total 250.29 last 25.64) - open/close 6643 (time 1788907067.61 total 261.24 last 26.51) - open/close 6969 (time 1788907077.61 total 271.24 last 32.58) - open/close 7196 (time 1788907087.62 total 281.25 last 22.69) - open/close 7516 (time 1788907097.62 total 291.25 last 32.00) - open/close 7786 (time 1788907107.93 total 301.56 last 26.19) - open/close 8047 (time 1788907118.04 total 311.66 last 25.82) - open/close 8323 (time 1788907128.47 total 322.10 last 26.45) - open/close 8708 (time 1788907139.04 total 332.67 last 36.44) - open/close 8887 (time 1788907150.11 total 343.74 last 16.17) - open/close 9223 (time 1788907160.41 total 354.03 last 32.63) - open/close 9345 (time 1788907171.80 total 365.43 last 10.71) - open/close 9675 (time 1788907182.23 total 375.86 last 31.64) - open/close 9943 (time 1788907192.23 total 385.86 last 26.79) - open/close 10000 (time 1788907194.74 total 388.37 last 22.79) - open/close 10246 (time 1788907204.74 total 398.37 last 24.59) - open/close 10533 (time 1788907215.76 total 409.39 last 26.03) - open/close 10806 (time 1788907225.82 total 419.44 last 27.16) - open/close 10973 (time 1788907237.48 total 431.11 last 14.32) - open/close 11245 (time 1788907247.96 total 441.59 last 25.96) - open/close 11453 (time 1788907257.96 total 451.59 last 20.80) - open/close 11801 (time 1788907267.97 total 461.60 last 34.77) - open/close 11956 (time 1788907278.30 total 471.93 last 14.99) - open/close 12171 (time 1788907289.19 total 482.82 last 19.76) - open/close 12375 (time 1788907301.86 total 495.49 last 16.10) - open/close 12572 (time 1788907311.88 total 505.51 last 19.65) - open/close 13079 (time 1788907321.88 total 515.51 last 50.70) - open/close 13729 (time 1788907331.89 total 525.52 last 64.99) - open/close 14471 (time 1788907341.90 total 535.53 last 74.10) - open/close 15130 (time 1788907351.91 total 545.54 last 65.86) - open/close 15815 (time 1788907361.91 total 555.54 last 68.45) - open/close 16689 (time 1788907371.92 total 565.55 last 87.34) - open/close 17428 (time 1788907381.92 total 575.55 last 73.89) - open/close 18276 (time 1788907391.93 total 585.56 last 84.76) - open/close 19212 (time 1788907401.94 total 595.57 last 93.52) - open/close 20000 (time 1788907410.74 total 604.37 last 89.48) - open/close 20837 (time 1788907420.74 total 614.37 last 83.69) - open/close 21687 (time 1788907430.75 total 624.38 last 84.94) - open/close 22562 (time 1788907440.77 total 634.40 last 87.36) - open/close 23471 (time 1788907450.78 total 644.41 last 90.79) - open/close 24404 (time 1788907460.79 total 654.42 last 93.22) - open/close 25227 (time 1788907470.80 total 664.43 last 82.19) - open/close 26027 (time 1788907480.81 total 674.44 last 79.95) - open/close 26807 (time 1788907490.82 total 684.45 last 77.90) - open/close 27584 (time 1788907500.83 total 694.46 last 77.63) - open/close 28344 (time 1788907510.84 total 704.47 last 75.93) - open/close 29106 (time 1788907520.85 total 714.48 last 76.10) - open/close 29860 (time 1788907530.87 total 724.50 last 75.29) - open/close 30621 (time 1788907540.88 total 734.51 last 75.99) - open/close 31357 (time 1788907550.90 total 744.53 last 73.50) - open/close 32148 (time 1788907560.91 total 754.54 last 78.98) - open/close 32975 (time 1788907570.92 total 764.55 last 82.60) - open/close 33828 (time 1788907580.93 total 774.56 last 85.27) - open/close 34830 (time 1788907590.93 total 784.56 last 100.18) - open/close 35749 (time 1788907600.94 total 794.57 last 91.82) - open/close 36778 (time 1788907610.95 total 804.58 last 102.81) - open/close 37983 (time 1788907620.95 total 814.58 last 120.48) - open/close 39261 (time 1788907630.95 total 824.58 last 127.76) - open/close 40000 (time 1788907637.66 total 831.29 last 110.22) - open/close 41082 (time 1788907647.66 total 841.29 last 108.16) - open/close 42477 (time 1788907657.67 total 851.30 last 139.33) - open/close 45320 (time 1788907667.68 total 861.31 last 284.17) - open/close 47767 (time 1788907677.68 total 871.31 last 244.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 879.34 seconds: 56.44 ops/second - unlinked 0 (time 1788907687 ; total 0 ; last 0) - unlinked 10000 (time 1788907726 ; total 39 ; last 39) - unlinked 20000 (time 1788907747 ; total 60 ; last 21) - unlinked 30000 (time 1788907774 ; total 87 ; last 27) - unlinked 40000 (time 1788907806 ; total 119 ; last 32) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 148 seconds: 335.344604 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2