== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:42:13 (1786833733) start mds service on oleg113-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg113-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg113-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg113-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 136 (time 1786833864.91 total 10.63 last 12.80) - open/close 241 (time 1786833876.16 total 21.87 last 9.34) - open/close 410 (time 1786833886.16 total 31.87 last 16.89) - open/close 563 (time 1786833896.73 total 42.44 last 14.48) - open/close 681 (time 1786833907.24 total 52.95 last 11.23) - open/close 943 (time 1786833920.17 total 65.89 last 20.25) - open/close 1118 (time 1786833930.54 total 76.25 last 16.89) - open/close 1338 (time 1786833940.90 total 86.62 last 21.22) - open/close 1577 (time 1786833950.91 total 96.62 last 23.89) - open/close 1809 (time 1786833961.25 total 106.96 last 22.43) - open/close 2065 (time 1786833971.50 total 117.21 last 24.98) - open/close 2292 (time 1786833982.44 total 128.15 last 20.74) - open/close 2494 (time 1786833992.50 total 138.21 last 20.08) - open/close 2777 (time 1786834002.50 total 148.21 last 28.30) - open/close 3030 (time 1786834013.17 total 158.88 last 23.71) - open/close 3234 (time 1786834023.98 total 169.69 last 18.88) - open/close 3384 (time 1786834034.48 total 180.20 last 14.28) - open/close 3563 (time 1786834044.73 total 190.45 last 17.46) - open/close 3781 (time 1786834054.74 total 200.45 last 21.79) - open/close 3999 (time 1786834065.06 total 210.78 last 21.11) - open/close 4236 (time 1786834077.49 total 223.21 last 19.07) - open/close 4528 (time 1786834090.54 total 236.26 last 22.38) - open/close 4789 (time 1786834101.41 total 247.13 last 24.01) - open/close 5072 (time 1786834111.42 total 257.13 last 28.28) - open/close 5271 (time 1786834121.43 total 267.14 last 19.89) - open/close 5481 (time 1786834131.46 total 277.17 last 20.94) - open/close 5718 (time 1786834142.06 total 287.78 last 22.34) - open/close 5924 (time 1786834152.55 total 298.27 last 19.64) - open/close 6199 (time 1786834162.56 total 308.27 last 27.49) - open/close 6365 (time 1786834172.81 total 318.53 last 16.19) - open/close 6587 (time 1786834182.83 total 328.54 last 22.17) - open/close 6825 (time 1786834192.84 total 338.55 last 23.77) - open/close 7046 (time 1786834202.85 total 348.56 last 22.07) - open/close 7308 (time 1786834213.69 total 359.40 last 24.17) - open/close 7459 (time 1786834223.88 total 369.59 last 14.82) - open/close 7761 (time 1786834233.97 total 379.68 last 29.95) - open/close 7908 (time 1786834243.99 total 389.70 last 14.66) - open/close 8160 (time 1786834254.50 total 400.22 last 23.97) - open/close 8519 (time 1786834264.52 total 410.23 last 35.86) - open/close 8797 (time 1786834274.52 total 420.24 last 27.78) - open/close 9088 (time 1786834284.53 total 430.24 last 29.09) - open/close 9281 (time 1786834295.28 total 440.99 last 17.95) - open/close 9568 (time 1786834305.28 total 451.00 last 28.68) - open/close 9757 (time 1786834315.29 total 461.01 last 18.88) - open/close 10000 (time 1786834325.00 total 470.72 last 25.03) - open/close 10239 (time 1786834335.02 total 480.73 last 23.86) - open/close 10443 (time 1786834345.93 total 491.64 last 18.70) - open/close 10690 (time 1786834355.94 total 501.65 last 24.67) - open/close 10991 (time 1786834365.94 total 511.66 last 30.08) - open/close 11235 (time 1786834375.96 total 521.67 last 24.37) - open/close 11508 (time 1786834386.59 total 532.30 last 25.67) - open/close 11759 (time 1786834396.61 total 542.33 last 25.04) - open/close 11996 (time 1786834406.88 total 552.60 last 23.08) - open/close 12257 (time 1786834417.57 total 563.29 last 24.41) - open/close 12422 (time 1786834427.59 total 573.31 last 16.47) - open/close 12679 (time 1786834439.28 total 584.99 last 21.99) - open/close 12870 (time 1786834449.98 total 595.70 last 17.84) - open/close 13173 (time 1786834460.34 total 606.05 last 29.27) - open/close 13453 (time 1786834470.78 total 616.49 last 26.81) - open/close 13635 (time 1786834480.87 total 626.59 last 18.03) - open/close 13898 (time 1786834491.45 total 637.16 last 24.86) - open/close 14151 (time 1786834502.18 total 647.90 last 23.58) - open/close 14324 (time 1786834513.07 total 658.79 last 15.89) - open/close 14560 (time 1786834523.12 total 668.83 last 23.50) - open/close 14821 (time 1786834533.44 total 679.16 last 25.28) - open/close 15031 (time 1786834543.44 total 689.16 last 21.00) - open/close 15273 (time 1786834553.46 total 699.17 last 24.17) - open/close 15528 (time 1786834563.82 total 709.54 last 24.60) - open/close 15821 (time 1786834574.91 total 720.62 last 26.43) - open/close 16002 (time 1786834585.15 total 730.86 last 17.67) - open/close 16261 (time 1786834596.14 total 741.86 last 23.56) - open/close 16526 (time 1786834606.15 total 751.86 last 26.49) - open/close 16776 (time 1786834616.15 total 761.87 last 24.99) - open/close 17017 (time 1786834626.35 total 772.06 last 23.64) - open/close 17271 (time 1786834636.35 total 782.06 last 25.39) - open/close 17527 (time 1786834646.36 total 792.08 last 25.57) - open/close 17728 (time 1786834656.43 total 802.15 last 19.96) - open/close 17953 (time 1786834666.45 total 812.16 last 22.47) - open/close 18209 (time 1786834676.45 total 822.17 last 25.58) - open/close 18507 (time 1786834686.45 total 832.17 last 29.80) - open/close 18704 (time 1786834696.46 total 842.18 last 19.68) - open/close 19010 (time 1786834707.52 total 853.23 last 27.68) - open/close 19291 (time 1786834718.50 total 864.21 last 25.59) - open/close 19586 (time 1786834729.46 total 875.17 last 26.92) - open/close 19864 (time 1786834741.30 total 887.01 last 23.48) - open/close 20000 (time 1786834750.90 total 896.61 last 14.17) - open/close 20289 (time 1786834760.91 total 906.63 last 28.86) - open/close 20545 (time 1786834771.57 total 917.28 last 24.02) - open/close 21508 (time 1786834781.57 total 927.29 last 96.27) - open/close 22414 (time 1786834791.58 total 937.30 last 90.51) - open/close 23393 (time 1786834801.60 total 947.31 last 97.75) - open/close 24300 (time 1786834811.60 total 957.32 last 90.63) - open/close 25230 (time 1786834821.60 total 967.32 last 93.00) - open/close 26090 (time 1786834831.61 total 977.33 last 85.94) - open/close 27104 (time 1786834841.62 total 987.34 last 101.27) - open/close 28129 (time 1786834851.64 total 997.35 last 102.38) - open/close 29111 (time 1786834861.64 total 1007.35 last 98.18) - open/close 30000 (time 1786834871.61 total 1017.33 last 89.11) - open/close 30848 (time 1786834881.62 total 1027.33 last 84.76) - open/close 31815 (time 1786834891.62 total 1037.34 last 96.66) - open/close 32808 (time 1786834901.63 total 1047.34 last 99.25) - open/close 33785 (time 1786834911.63 total 1057.35 last 97.66) - open/close 34662 (time 1786834921.64 total 1067.36 last 87.60) - open/close 35627 (time 1786834931.65 total 1077.36 last 96.45) - open/close 36463 (time 1786834941.65 total 1087.36 last 83.60) - open/close 37427 (time 1786834951.65 total 1097.37 last 96.38) - open/close 38416 (time 1786834961.66 total 1107.37 last 98.85) - open/close 39315 (time 1786834971.67 total 1117.38 last 89.82) - open/close 40000 (time 1786834978.04 total 1123.76 last 107.41) - open/close 41023 (time 1786834988.05 total 1133.76 last 102.28) - open/close 41996 (time 1786834998.05 total 1143.77 last 97.25) - open/close 42904 (time 1786835008.06 total 1153.78 last 90.70) - open/close 43792 (time 1786835018.07 total 1163.78 last 88.73) - open/close 44609 (time 1786835028.08 total 1173.80 last 81.61) - open/close 45552 (time 1786835038.08 total 1183.80 last 94.27) - open/close 46539 (time 1786835048.09 total 1193.80 last 98.67) - open/close 47536 (time 1786835058.10 total 1203.81 last 99.60) - open/close 48610 (time 1786835068.10 total 1213.81 last 107.39) - open/close 49595 (time 1786835078.11 total 1223.82 last 98.41) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1224.19 seconds: 40.54 ops/second - unlinked 0 (time 1786835082 ; total 0 ; last 0) - unlinked 10000 (time 1786835185 ; total 103 ; last 103) - unlinked 20000 (time 1786835274 ; total 192 ; last 89) - unlinked 30000 (time 1786835336 ; total 254 ; last 62) - unlinked 40000 (time 1786835400 ; total 318 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 378 seconds: 131.298935 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: modules unloaded. pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2