== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:51:24 (1787388684) start mds service on oleg428-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg428-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg428-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg428-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1787388785.21 total 10.59 last 23.99) - open/close 464 (time 1787388795.27 total 20.65 last 20.87) - open/close 689 (time 1787388810.37 total 35.74 last 14.91) - open/close 898 (time 1787388820.38 total 45.75 last 20.88) - open/close 1059 (time 1787388830.40 total 55.78 last 16.06) - open/close 1338 (time 1787388840.56 total 65.94 last 27.46) - open/close 1591 (time 1787388850.75 total 76.12 last 24.84) - open/close 1872 (time 1787388860.76 total 86.14 last 28.07) - open/close 2135 (time 1787388871.82 total 97.19 last 23.79) - open/close 2326 (time 1787388881.84 total 107.21 last 19.06) - open/close 2575 (time 1787388892.02 total 117.40 last 24.44) - open/close 2870 (time 1787388902.97 total 128.35 last 26.94) - open/close 3157 (time 1787388913.65 total 139.03 last 26.88) - open/close 3441 (time 1787388923.66 total 149.04 last 28.37) - open/close 3742 (time 1787388933.67 total 159.05 last 30.07) - open/close 3990 (time 1787388944.63 total 170.00 last 22.64) - open/close 4262 (time 1787388955.26 total 180.64 last 25.57) - open/close 4536 (time 1787388966.09 total 191.46 last 25.32) - open/close 4699 (time 1787388976.09 total 201.47 last 16.29) - open/close 4856 (time 1787388986.88 total 212.26 last 14.55) - open/close 4967 (time 1787388997.25 total 222.63 last 10.71) - open/close 5145 (time 1787389007.35 total 232.72 last 17.63) - open/close 5247 (time 1787389017.67 total 243.05 last 9.88) - open/close 5456 (time 1787389027.69 total 253.06 last 20.88) - open/close 5675 (time 1787389037.71 total 263.09 last 21.84) - open/close 5942 (time 1787389050.56 total 275.94 last 20.79) - open/close 6142 (time 1787389061.24 total 286.61 last 18.73) - open/close 6350 (time 1787389072.07 total 297.45 last 19.19) - open/close 6641 (time 1787389082.12 total 307.49 last 28.98) - open/close 6973 (time 1787389092.75 total 318.13 last 31.22) - open/close 7229 (time 1787389102.75 total 328.13 last 25.60) - open/close 7489 (time 1787389113.34 total 338.71 last 24.56) - open/close 7749 (time 1787389123.76 total 349.14 last 24.94) - open/close 8015 (time 1787389134.09 total 359.46 last 25.77) - open/close 8234 (time 1787389145.16 total 370.54 last 19.78) - open/close 8550 (time 1787389155.39 total 380.77 last 30.90) - open/close 8809 (time 1787389166.28 total 391.66 last 23.77) - open/close 9043 (time 1787389176.82 total 402.20 last 22.21) - open/close 9250 (time 1787389186.83 total 412.20 last 20.69) - open/close 9462 (time 1787389196.83 total 422.20 last 21.20) - open/close 9666 (time 1787389206.84 total 432.21 last 20.38) - open/close 9896 (time 1787389216.84 total 442.22 last 22.99) - open/close 10000 (time 1787389224.81 total 450.19 last 13.04) - open/close 10230 (time 1787389235.38 total 460.76 last 21.76) - open/close 10502 (time 1787389245.39 total 470.77 last 27.17) - open/close 10767 (time 1787389255.50 total 480.87 last 26.23) - open/close 11058 (time 1787389265.53 total 490.91 last 29.00) - open/close 11315 (time 1787389276.09 total 501.46 last 24.35) - open/close 11628 (time 1787389287.10 total 512.47 last 28.43) - open/close 11787 (time 1787389297.49 total 522.86 last 15.31) - open/close 12009 (time 1787389307.49 total 532.87 last 22.19) - open/close 12258 (time 1787389317.49 total 542.87 last 24.89) - open/close 12521 (time 1787389327.50 total 552.87 last 26.29) - open/close 12853 (time 1787389337.51 total 562.89 last 33.15) - open/close 13023 (time 1787389348.62 total 574.00 last 15.30) - open/close 13321 (time 1787389358.63 total 584.01 last 29.77) - open/close 13519 (time 1787389369.27 total 594.64 last 18.62) - open/close 13802 (time 1787389379.71 total 605.09 last 27.09) - open/close 14048 (time 1787389389.72 total 615.09 last 24.59) - open/close 14314 (time 1787389400.32 total 625.70 last 25.08) - open/close 14583 (time 1787389410.94 total 636.32 last 25.32) - open/close 14879 (time 1787389421.45 total 646.83 last 28.17) - open/close 15147 (time 1787389431.74 total 657.11 last 26.06) - open/close 15433 (time 1787389442.17 total 667.55 last 27.41) - open/close 15722 (time 1787389452.74 total 678.11 last 27.36) - open/close 15945 (time 1787389462.85 total 688.23 last 22.04) - open/close 16198 (time 1787389473.21 total 698.59 last 24.42) - open/close 16475 (time 1787389483.22 total 708.60 last 27.68) - open/close 16648 (time 1787389494.15 total 719.52 last 15.83) - open/close 16786 (time 1787389504.62 total 730.00 last 13.17) - open/close 17053 (time 1787389514.68 total 740.05 last 26.56) - open/close 17298 (time 1787389525.32 total 750.69 last 23.03) - open/close 17581 (time 1787389535.81 total 761.19 last 26.97) - open/close 17817 (time 1787389545.84 total 771.22 last 23.52) - open/close 18047 (time 1787389555.85 total 781.22 last 22.99) - open/close 18229 (time 1787389566.13 total 791.51 last 17.70) - open/close 18509 (time 1787389577.09 total 802.47 last 25.55) - open/close 18759 (time 1787389587.46 total 812.84 last 24.11) - open/close 19041 (time 1787389597.47 total 822.84 last 28.19) - open/close 19324 (time 1787389607.47 total 832.85 last 28.29) - open/close 19619 (time 1787389618.50 total 843.87 last 26.76) - open/close 19908 (time 1787389628.50 total 853.87 last 28.90) - open/close 20000 (time 1787389632.02 total 857.39 last 26.14) - open/close 20275 (time 1787389642.02 total 867.39 last 27.50) - open/close 20532 (time 1787389652.21 total 877.59 last 25.21) - open/close 20811 (time 1787389662.22 total 887.60 last 27.88) - open/close 21058 (time 1787389672.39 total 897.76 last 24.30) - open/close 21290 (time 1787389683.32 total 908.70 last 21.21) - open/close 21448 (time 1787389693.34 total 918.72 last 15.77) - open/close 21622 (time 1787389703.37 total 928.75 last 17.35) - open/close 22045 (time 1787389713.38 total 938.75 last 42.28) - open/close 22958 (time 1787389723.38 total 948.76 last 91.23) - open/close 23845 (time 1787389733.39 total 958.76 last 88.68) - open/close 24673 (time 1787389743.39 total 968.77 last 82.74) - open/close 25472 (time 1787389753.40 total 978.78 last 79.85) - open/close 26295 (time 1787389763.40 total 988.78 last 82.29) - open/close 27127 (time 1787389773.40 total 998.78 last 83.17) - open/close 27979 (time 1787389783.41 total 1008.79 last 85.16) - open/close 28813 (time 1787389793.42 total 1018.80 last 83.32) - open/close 29668 (time 1787389803.42 total 1028.80 last 85.49) - open/close 30000 (time 1787389807.01 total 1032.39 last 92.39) - open/close 30880 (time 1787389817.03 total 1042.40 last 87.87) - open/close 31761 (time 1787389827.04 total 1052.42 last 88.01) - open/close 32552 (time 1787389837.04 total 1062.42 last 79.05) - open/close 33331 (time 1787389847.05 total 1072.42 last 77.89) - open/close 34090 (time 1787389857.05 total 1082.42 last 75.89) - open/close 34944 (time 1787389867.05 total 1092.42 last 85.40) - open/close 35774 (time 1787389877.05 total 1102.43 last 82.94) - open/close 36710 (time 1787389887.06 total 1112.44 last 93.55) - open/close 37583 (time 1787389897.06 total 1122.44 last 87.27) - open/close 38426 (time 1787389907.08 total 1132.46 last 84.15) - open/close 39237 (time 1787389917.08 total 1142.46 last 81.10) - open/close 40000 (time 1787389925.83 total 1151.21 last 87.20) - open/close 40678 (time 1787389935.84 total 1161.21 last 67.76) - open/close 41352 (time 1787389945.85 total 1171.23 last 67.31) - open/close 42102 (time 1787389955.85 total 1181.23 last 74.99) - open/close 42935 (time 1787389965.85 total 1191.23 last 83.28) - open/close 43752 (time 1787389975.86 total 1201.23 last 81.67) - open/close 44510 (time 1787389985.87 total 1211.24 last 75.70) - open/close 45350 (time 1787389995.87 total 1221.25 last 83.97) - open/close 46171 (time 1787390005.88 total 1231.26 last 82.01) - open/close 46976 (time 1787390015.89 total 1241.27 last 80.40) - open/close 47710 (time 1787390025.92 total 1251.30 last 73.21) - open/close 48377 (time 1787390035.93 total 1261.30 last 66.66) - open/close 49102 (time 1787390045.93 total 1271.31 last 72.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1278.77 seconds: 38.81 ops/second - unlinked 0 (time 1787390057 ; total 0 ; last 0) - unlinked 10000 (time 1787390184 ; total 127 ; last 127) - unlinked 20000 (time 1787390312 ; total 255 ; last 128) - unlinked 30000 (time 1787390441 ; total 384 ; last 129) - unlinked 40000 (time 1787390569 ; total 512 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 646 seconds: 76.828171 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local oleg428-server: modules unloaded. pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2