== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:43:10 (1781242990) start mds service on oleg114-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg114-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1781243087.26 total 10.72 last 20.70) - open/close 433 (time 1781243097.27 total 20.73 last 21.08) - open/close 677 (time 1781243109.29 total 32.75 last 20.29) - open/close 835 (time 1781243119.51 total 42.97 last 15.46) - open/close 958 (time 1781243133.35 total 56.81 last 8.89) - open/close 1170 (time 1781243143.46 total 66.92 last 20.97) - open/close 1408 (time 1781243153.46 total 76.92 last 23.78) - open/close 1663 (time 1781243163.47 total 86.93 last 25.48) - open/close 1883 (time 1781243174.37 total 97.83 last 20.18) - open/close 2128 (time 1781243184.38 total 107.84 last 24.48) - open/close 2364 (time 1781243194.78 total 118.24 last 22.69) - open/close 2624 (time 1781243205.46 total 128.92 last 24.34) - open/close 2868 (time 1781243215.92 total 139.38 last 23.34) - open/close 3101 (time 1781243226.91 total 150.37 last 21.20) - open/close 3367 (time 1781243237.73 total 161.19 last 24.58) - open/close 3638 (time 1781243248.43 total 171.89 last 25.33) - open/close 3902 (time 1781243258.43 total 181.90 last 26.38) - open/close 4139 (time 1781243269.27 total 192.73 last 21.87) - open/close 4367 (time 1781243279.65 total 203.11 last 21.98) - open/close 4604 (time 1781243289.65 total 213.11 last 23.69) - open/close 4837 (time 1781243299.66 total 223.12 last 23.27) - open/close 5033 (time 1781243310.25 total 233.71 last 18.51) - open/close 5256 (time 1781243320.26 total 243.72 last 22.28) - open/close 5469 (time 1781243330.60 total 254.06 last 20.59) - open/close 5683 (time 1781243340.60 total 264.06 last 21.40) - open/close 5884 (time 1781243351.21 total 274.67 last 18.95) - open/close 6102 (time 1781243361.31 total 284.77 last 21.58) - open/close 6358 (time 1781243371.50 total 294.96 last 25.13) - open/close 6611 (time 1781243381.50 total 304.96 last 25.29) - open/close 6884 (time 1781243391.57 total 315.03 last 27.10) - open/close 7130 (time 1781243401.58 total 325.04 last 24.59) - open/close 7311 (time 1781243411.62 total 335.08 last 18.02) - open/close 7546 (time 1781243422.30 total 345.76 last 22.00) - open/close 7805 (time 1781243433.19 total 356.65 last 23.78) - open/close 8113 (time 1781243444.13 total 367.59 last 28.17) - open/close 8358 (time 1781243454.70 total 378.16 last 23.16) - open/close 8594 (time 1781243465.70 total 389.16 last 21.46) - open/close 8834 (time 1781243475.72 total 399.18 last 23.95) - open/close 9068 (time 1781243486.25 total 409.71 last 22.23) - open/close 9300 (time 1781243496.42 total 419.89 last 22.80) - open/close 9594 (time 1781243506.44 total 429.90 last 29.36) - open/close 9853 (time 1781243516.46 total 439.92 last 25.85) - open/close 10000 (time 1781243521.85 total 445.31 last 27.27) - open/close 10211 (time 1781243531.89 total 455.35 last 21.02) - open/close 10476 (time 1781243541.89 total 465.36 last 26.48) - open/close 10634 (time 1781243552.35 total 475.81 last 15.12) - open/close 10920 (time 1781243562.36 total 485.82 last 28.56) - open/close 11160 (time 1781243573.02 total 496.48 last 22.52) - open/close 11377 (time 1781243583.89 total 507.36 last 19.95) - open/close 11590 (time 1781243594.08 total 517.54 last 20.91) - open/close 11890 (time 1781243605.08 total 528.55 last 27.27) - open/close 12094 (time 1781243615.29 total 538.75 last 20.00) - open/close 12382 (time 1781243625.57 total 549.03 last 28.02) - open/close 12622 (time 1781243635.57 total 559.03 last 23.99) - open/close 12875 (time 1781243646.52 total 569.98 last 23.11) - open/close 13095 (time 1781243656.75 total 580.21 last 21.50) - open/close 13323 (time 1781243667.03 total 590.49 last 22.17) - open/close 13457 (time 1781243677.15 total 600.61 last 13.24) - open/close 13662 (time 1781243688.00 total 611.46 last 18.90) - open/close 13847 (time 1781243698.85 total 622.31 last 17.05) - open/close 14073 (time 1781243708.85 total 632.31 last 22.60) - open/close 14325 (time 1781243719.95 total 643.41 last 22.69) - open/close 14467 (time 1781243729.97 total 653.43 last 14.17) - open/close 14629 (time 1781243740.66 total 664.12 last 15.16) - open/close 14729 (time 1781243751.33 total 674.80 last 9.37) - open/close 14935 (time 1781243761.34 total 684.80 last 20.60) - open/close 15159 (time 1781243771.36 total 694.82 last 22.35) - open/close 15300 (time 1781243781.38 total 704.85 last 14.06) - open/close 15467 (time 1781243792.23 total 715.70 last 15.39) - open/close 15607 (time 1781243803.04 total 726.50 last 12.95) - open/close 15801 (time 1781243813.86 total 737.33 last 17.92) - open/close 16028 (time 1781243824.12 total 747.58 last 22.13) - open/close 16207 (time 1781243834.12 total 757.58 last 17.90) - open/close 16401 (time 1781243844.13 total 767.59 last 19.39) - open/close 16538 (time 1781243854.57 total 778.03 last 13.13) - open/close 16733 (time 1781243864.58 total 788.04 last 19.47) - open/close 16974 (time 1781243874.72 total 798.18 last 23.77) - open/close 17229 (time 1781243884.73 total 808.19 last 25.47) - open/close 17371 (time 1781243894.95 total 818.41 last 13.90) - open/close 17555 (time 1781243904.97 total 828.43 last 18.35) - open/close 17654 (time 1781243915.49 total 838.95 last 9.41) - open/close 17862 (time 1781243925.74 total 849.21 last 20.28) - open/close 18076 (time 1781243935.75 total 859.22 last 21.38) - open/close 18888 (time 1781243945.77 total 869.23 last 81.08) - open/close 19523 (time 1781243955.78 total 879.24 last 63.45) - open/close 20000 (time 1781243963.67 total 887.13 last 60.47) - open/close 20759 (time 1781243973.67 total 897.13 last 75.86) - open/close 21776 (time 1781243983.67 total 907.14 last 101.68) - open/close 22633 (time 1781243993.68 total 917.14 last 85.68) - open/close 23367 (time 1781244003.69 total 927.15 last 73.34) - open/close 24315 (time 1781244013.69 total 937.15 last 94.73) - open/close 25145 (time 1781244023.71 total 947.17 last 82.87) - open/close 26069 (time 1781244033.71 total 957.17 last 92.38) - open/close 26929 (time 1781244043.72 total 967.18 last 85.93) - open/close 27683 (time 1781244053.72 total 977.18 last 75.37) - open/close 28578 (time 1781244063.73 total 987.19 last 89.48) - open/close 29527 (time 1781244073.73 total 997.19 last 94.86) - open/close 30000 (time 1781244081.67 total 1005.13 last 59.60) - open/close 30703 (time 1781244091.67 total 1015.14 last 70.24) - open/close 31586 (time 1781244101.68 total 1025.14 last 88.26) - open/close 32544 (time 1781244111.68 total 1035.14 last 95.77) - open/close 33392 (time 1781244121.68 total 1045.14 last 84.80) - open/close 34254 (time 1781244131.69 total 1055.15 last 86.16) - open/close 35072 (time 1781244141.69 total 1065.15 last 81.80) - open/close 35928 (time 1781244151.69 total 1075.15 last 85.57) - open/close 36747 (time 1781244161.70 total 1085.16 last 81.83) - open/close 37667 (time 1781244171.71 total 1095.17 last 91.95) - open/close 38551 (time 1781244181.71 total 1105.17 last 88.37) - open/close 39497 (time 1781244191.73 total 1115.19 last 94.45) - open/close 40000 (time 1781244198.01 total 1121.47 last 80.06) - open/close 40593 (time 1781244208.05 total 1131.51 last 59.04) - open/close 41451 (time 1781244218.06 total 1141.52 last 85.74) - open/close 42243 (time 1781244228.06 total 1151.52 last 79.17) - open/close 43199 (time 1781244238.06 total 1161.53 last 95.58) - open/close 44227 (time 1781244248.07 total 1171.53 last 102.74) - open/close 45176 (time 1781244258.07 total 1181.53 last 94.90) - open/close 46023 (time 1781244268.08 total 1191.54 last 84.65) - open/close 46914 (time 1781244278.08 total 1201.54 last 89.09) - open/close 47920 (time 1781244288.08 total 1211.54 last 100.55) - open/close 48732 (time 1781244298.09 total 1221.55 last 81.15) - open/close 49537 (time 1781244308.10 total 1231.56 last 80.42) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1232.64 seconds: 40.26 ops/second - unlinked 0 (time 1781244313 ; total 0 ; last 0) - unlinked 10000 (time 1781244432 ; total 119 ; last 119) - unlinked 20000 (time 1781244557 ; total 244 ; last 125) - unlinked 30000 (time 1781244625 ; total 312 ; last 68) - unlinked 40000 (time 1781244699 ; total 386 ; last 74) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 453 seconds: 109.560707 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: modules unloaded. pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2