== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:28:28 (1788157708) start mds service on oleg645-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg645-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1788157812.96 total 10.01 last 21.37) - open/close 437 (time 1788157823.97 total 21.03 last 20.25) - open/close 688 (time 1788157834.53 total 31.59 last 23.77) - open/close 931 (time 1788157844.53 total 41.59 last 24.30) - open/close 1181 (time 1788157854.83 total 51.88 last 24.29) - open/close 1356 (time 1788157865.76 total 62.81 last 16.01) - open/close 1589 (time 1788157876.19 total 73.24 last 22.34) - open/close 1888 (time 1788157886.20 total 83.25 last 29.87) - open/close 2182 (time 1788157896.20 total 93.26 last 29.39) - open/close 2412 (time 1788157906.23 total 103.28 last 22.95) - open/close 2689 (time 1788157916.91 total 113.96 last 25.93) - open/close 2918 (time 1788157928.10 total 125.15 last 20.47) - open/close 3023 (time 1788157938.53 total 135.58 last 10.06) - open/close 3243 (time 1788157951.66 total 148.72 last 16.75) - open/close 3475 (time 1788157961.89 total 158.95 last 22.68) - open/close 3731 (time 1788157972.59 total 169.64 last 23.94) - open/close 4047 (time 1788157982.98 total 180.04 last 30.39) - open/close 4356 (time 1788157992.99 total 190.04 last 30.89) - open/close 4645 (time 1788158003.64 total 200.69 last 27.13) - open/close 4954 (time 1788158013.68 total 210.73 last 30.79) - open/close 5218 (time 1788158024.03 total 221.09 last 25.49) - open/close 5483 (time 1788158034.30 total 231.35 last 25.82) - open/close 5799 (time 1788158045.09 total 242.14 last 29.28) - open/close 6069 (time 1788158056.04 total 253.09 last 24.66) - open/close 6298 (time 1788158066.73 total 263.79 last 21.42) - open/close 6625 (time 1788158076.73 total 273.79 last 32.70) - open/close 6874 (time 1788158086.74 total 283.79 last 24.89) - open/close 7106 (time 1788158097.70 total 294.75 last 21.16) - open/close 7365 (time 1788158107.70 total 304.76 last 25.89) - open/close 7664 (time 1788158117.71 total 314.76 last 29.89) - open/close 7921 (time 1788158129.89 total 326.94 last 21.09) - open/close 8093 (time 1788158140.07 total 337.13 last 16.89) - open/close 8180 (time 1788158150.46 total 347.52 last 8.37) - open/close 8389 (time 1788158160.48 total 357.53 last 20.87) - open/close 8631 (time 1788158170.54 total 367.60 last 24.03) - open/close 8931 (time 1788158180.72 total 377.77 last 29.49) - open/close 9259 (time 1788158191.04 total 388.09 last 31.79) - open/close 9480 (time 1788158201.65 total 398.70 last 20.83) - open/close 9791 (time 1788158211.66 total 408.72 last 31.05) - open/close 10000 (time 1788158219.13 total 416.19 last 27.98) - open/close 10262 (time 1788158229.17 total 426.23 last 26.10) - open/close 10528 (time 1788158239.19 total 436.24 last 26.56) - open/close 10799 (time 1788158249.19 total 446.25 last 27.08) - open/close 11081 (time 1788158259.70 total 456.75 last 26.84) - open/close 11366 (time 1788158269.86 total 466.91 last 28.05) - open/close 11592 (time 1788158280.74 total 477.79 last 20.78) - open/close 11876 (time 1788158291.76 total 488.81 last 25.78) - open/close 12161 (time 1788158302.25 total 499.30 last 27.17) - open/close 12418 (time 1788158312.56 total 509.61 last 24.92) - open/close 12706 (time 1788158323.18 total 520.23 last 27.13) - open/close 12985 (time 1788158333.42 total 530.47 last 27.23) - open/close 13268 (time 1788158344.10 total 541.16 last 26.49) - open/close 13512 (time 1788158354.41 total 551.47 last 23.67) - open/close 13726 (time 1788158365.23 total 562.29 last 19.78) - open/close 14022 (time 1788158375.97 total 573.02 last 27.57) - open/close 14261 (time 1788158386.42 total 583.47 last 22.87) - open/close 14566 (time 1788158397.41 total 594.46 last 27.76) - open/close 14813 (time 1788158407.98 total 605.03 last 23.37) - open/close 15076 (time 1788158417.99 total 615.04 last 26.27) - open/close 15377 (time 1788158428.78 total 625.83 last 27.90) - open/close 15652 (time 1788158441.42 total 638.47 last 21.75) - open/close 15951 (time 1788158451.77 total 648.82 last 28.89) - open/close 16230 (time 1788158462.45 total 659.50 last 26.13) - open/close 16544 (time 1788158472.47 total 669.52 last 31.34) - open/close 16823 (time 1788158482.48 total 679.54 last 27.86) - open/close 17056 (time 1788158492.79 total 689.84 last 22.60) - open/close 17350 (time 1788158502.79 total 699.84 last 29.39) - open/close 17571 (time 1788158512.81 total 709.86 last 22.06) - open/close 17865 (time 1788158523.62 total 720.68 last 27.19) - open/close 18167 (time 1788158534.54 total 731.59 last 27.67) - open/close 18491 (time 1788158544.55 total 741.60 last 32.36) - open/close 18738 (time 1788158554.78 total 751.84 last 24.13) - open/close 19032 (time 1788158565.35 total 762.40 last 27.82) - open/close 19345 (time 1788158576.10 total 773.16 last 29.11) - open/close 19639 (time 1788158586.85 total 783.90 last 27.36) - open/close 19867 (time 1788158596.86 total 793.91 last 22.78) - open/close 20000 (time 1788158601.99 total 799.04 last 25.91) - open/close 20260 (time 1788158612.08 total 809.13 last 25.77) - open/close 20543 (time 1788158622.82 total 819.87 last 26.35) - open/close 20871 (time 1788158632.82 total 829.88 last 32.79) - open/close 21184 (time 1788158642.84 total 839.89 last 31.26) - open/close 21472 (time 1788158653.41 total 850.47 last 27.22) - open/close 21748 (time 1788158664.30 total 861.36 last 25.35) - open/close 22025 (time 1788158674.34 total 871.40 last 27.59) - open/close 22281 (time 1788158684.53 total 881.58 last 25.13) - open/close 22560 (time 1788158694.55 total 891.61 last 27.83) - open/close 22768 (time 1788158705.48 total 902.53 last 19.04) - open/close 23694 (time 1788158715.48 total 912.53 last 92.57) - open/close 24624 (time 1788158725.48 total 922.54 last 92.96) - open/close 25566 (time 1788158735.49 total 932.55 last 94.12) - open/close 26479 (time 1788158745.49 total 942.55 last 91.30) - open/close 27249 (time 1788158755.51 total 952.56 last 76.87) - open/close 28060 (time 1788158765.51 total 962.57 last 81.06) - open/close 28971 (time 1788158775.53 total 972.58 last 90.99) - open/close 29862 (time 1788158785.53 total 982.59 last 89.04) - open/close 30814 (time 1788158795.53 total 992.59 last 95.19) - open/close 31740 (time 1788158805.54 total 1002.59 last 92.53) - open/close 32685 (time 1788158815.55 total 1012.61 last 94.37) - open/close 33611 (time 1788158825.56 total 1022.61 last 92.59) - open/close 34507 (time 1788158835.57 total 1032.62 last 89.49) - open/close 35452 (time 1788158845.57 total 1042.63 last 94.44) - open/close 36399 (time 1788158855.57 total 1052.63 last 94.70) - open/close 37232 (time 1788158865.59 total 1062.64 last 83.17) - open/close 38068 (time 1788158875.59 total 1072.65 last 83.57) - open/close 38948 (time 1788158885.60 total 1082.65 last 87.96) - open/close 39871 (time 1788158895.60 total 1092.66 last 92.25) - open/close 40778 (time 1788158905.61 total 1102.66 last 90.63) - open/close 41655 (time 1788158915.62 total 1112.67 last 87.65) - open/close 42541 (time 1788158925.62 total 1122.67 last 88.58) - open/close 43325 (time 1788158935.64 total 1132.69 last 78.24) - open/close 44176 (time 1788158945.65 total 1142.70 last 85.05) - open/close 45154 (time 1788158955.66 total 1152.71 last 97.68) - open/close 46066 (time 1788158965.66 total 1162.72 last 91.15) - open/close 46961 (time 1788158975.68 total 1172.73 last 89.33) - open/close 47968 (time 1788158985.68 total 1182.74 last 100.68) - open/close 48839 (time 1788158995.69 total 1192.74 last 87.06) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1201.18 seconds: 41.32 ops/second - unlinked 0 (time 1788159007 ; total 0 ; last 0) - unlinked 10000 (time 1788159121 ; total 114 ; last 114) - unlinked 20000 (time 1788159235 ; total 228 ; last 114) - unlinked 30000 (time 1788159307 ; total 300 ; last 72) - unlinked 40000 (time 1788159375 ; total 368 ; last 68) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 428 seconds: 115.960281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2