== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:05:01 (1784124301) start mds service on oleg321-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg321-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg321-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg321-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1784124359.42 total 10.76 last 22.87) - open/close 458 (time 1784124369.97 total 21.31 last 20.09) - open/close 704 (time 1784124379.98 total 31.31 last 24.60) - open/close 935 (time 1784124390.90 total 42.24 last 21.14) - open/close 1179 (time 1784124401.72 total 53.06 last 22.55) - open/close 1409 (time 1784124412.47 total 63.81 last 21.40) - open/close 1659 (time 1784124422.48 total 73.82 last 24.97) - open/close 1836 (time 1784124432.95 total 84.29 last 16.90) - open/close 2101 (time 1784124442.95 total 94.29 last 26.49) - open/close 2340 (time 1784124453.81 total 105.15 last 22.01) - open/close 2682 (time 1784124464.44 total 115.77 last 32.19) - open/close 2979 (time 1784124474.88 total 126.22 last 28.44) - open/close 3164 (time 1784124485.06 total 136.40 last 18.17) - open/close 3293 (time 1784124495.68 total 147.02 last 12.15) - open/close 3480 (time 1784124506.40 total 157.74 last 17.45) - open/close 3648 (time 1784124516.42 total 167.76 last 16.77) - open/close 3866 (time 1784124526.43 total 177.76 last 21.78) - open/close 4028 (time 1784124536.43 total 187.76 last 16.20) - open/close 4230 (time 1784124546.82 total 198.16 last 19.44) - open/close 4454 (time 1784124557.00 total 208.33 last 22.01) - open/close 4614 (time 1784124567.10 total 218.44 last 15.84) - open/close 4736 (time 1784124577.86 total 229.20 last 11.33) - open/close 4901 (time 1784124587.90 total 239.23 last 16.45) - open/close 5024 (time 1784124598.40 total 249.73 last 11.71) - open/close 5138 (time 1784124608.96 total 260.29 last 10.80) - open/close 5271 (time 1784124620.67 total 272.01 last 11.35) - open/close 5450 (time 1784124630.78 total 282.12 last 17.70) - open/close 5597 (time 1784124641.60 total 292.94 last 13.59) - open/close 5784 (time 1784124652.03 total 303.37 last 17.92) - open/close 5916 (time 1784124662.85 total 314.19 last 12.20) - open/close 6073 (time 1784124672.95 total 324.29 last 15.55) - open/close 6214 (time 1784124683.03 total 334.37 last 13.99) - open/close 6376 (time 1784124693.37 total 344.71 last 15.67) - open/close 6547 (time 1784124703.74 total 355.07 last 16.50) - open/close 6689 (time 1784124714.70 total 366.04 last 12.95) - open/close 6839 (time 1784124724.99 total 376.33 last 14.58) - open/close 7021 (time 1784124735.61 total 386.95 last 17.14) - open/close 7289 (time 1784124745.62 total 396.96 last 26.77) - open/close 7446 (time 1784124756.55 total 407.88 last 14.37) - open/close 7577 (time 1784124767.55 total 418.89 last 11.90) - open/close 7767 (time 1784124777.57 total 428.91 last 18.96) - open/close 7953 (time 1784124787.85 total 439.19 last 18.09) - open/close 8212 (time 1784124798.60 total 449.94 last 24.10) - open/close 8444 (time 1784124808.70 total 460.04 last 22.97) - open/close 8710 (time 1784124819.02 total 470.36 last 25.77) - open/close 9014 (time 1784124829.03 total 480.37 last 30.36) - open/close 9110 (time 1784124839.48 total 490.82 last 9.19) - open/close 9265 (time 1784124849.72 total 501.06 last 15.13) - open/close 9391 (time 1784124860.60 total 511.94 last 11.59) - open/close 9507 (time 1784124871.17 total 522.51 last 10.97) - open/close 9647 (time 1784124882.18 total 533.52 last 12.72) - open/close 9854 (time 1784124892.86 total 544.20 last 19.38) - open/close 10000 (time 1784124899.51 total 550.85 last 21.95) - open/close 10147 (time 1784124910.47 total 561.80 last 13.42) - open/close 10258 (time 1784124920.95 total 572.29 last 10.59) - open/close 10394 (time 1784124931.71 total 583.04 last 12.64) - open/close 10525 (time 1784124944.83 total 596.17 last 9.98) - open/close 10667 (time 1784124955.52 total 606.86 last 13.28) - open/close 11649 (time 1784124965.52 total 616.86 last 98.18) - open/close 12794 (time 1784124975.53 total 626.87 last 114.42) - open/close 14031 (time 1784124985.54 total 636.87 last 123.64) - open/close 15273 (time 1784124995.54 total 646.88 last 124.11) - open/close 16646 (time 1784125005.55 total 656.89 last 137.20) - open/close 17999 (time 1784125015.56 total 666.89 last 135.25) - open/close 19074 (time 1784125025.56 total 676.90 last 107.46) - open/close 20000 (time 1784125033.65 total 684.98 last 114.48) - open/close 21113 (time 1784125043.65 total 694.99 last 111.25) - open/close 22158 (time 1784125053.65 total 704.99 last 104.48) - open/close 23275 (time 1784125063.66 total 715.00 last 111.64) - open/close 24292 (time 1784125073.67 total 725.01 last 101.55) - open/close 25376 (time 1784125083.69 total 735.03 last 108.24) - open/close 26499 (time 1784125093.69 total 745.03 last 112.29) - open/close 27606 (time 1784125103.69 total 755.03 last 110.69) - open/close 28741 (time 1784125113.70 total 765.03 last 113.43) - open/close 29835 (time 1784125123.70 total 775.04 last 109.39) - open/close 30954 (time 1784125133.70 total 785.04 last 111.85) - open/close 32115 (time 1784125143.71 total 795.04 last 116.05) - open/close 33409 (time 1784125153.71 total 805.05 last 129.38) - open/close 34639 (time 1784125163.71 total 815.05 last 122.98) - open/close 35837 (time 1784125173.71 total 825.05 last 119.76) - open/close 37086 (time 1784125183.71 total 835.05 last 124.90) - open/close 38191 (time 1784125193.72 total 845.06 last 110.45) - open/close 39327 (time 1784125203.72 total 855.06 last 113.60) - open/close 40000 (time 1784125210.15 total 861.49 last 104.58) - open/close 40996 (time 1784125220.17 total 871.50 last 99.48) - open/close 42126 (time 1784125230.17 total 881.51 last 112.94) - open/close 43195 (time 1784125240.18 total 891.51 last 106.85) - open/close 44418 (time 1784125250.18 total 901.52 last 122.27) - open/close 45618 (time 1784125260.19 total 911.52 last 119.90) - open/close 46822 (time 1784125270.19 total 921.53 last 120.36) - open/close 48091 (time 1784125280.19 total 931.53 last 126.87) - open/close 49490 (time 1784125290.20 total 941.53 last 139.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 942.54 seconds: 52.66 ops/second - unlinked 0 (time 1784125293 ; total 0 ; last 0) - unlinked 10000 (time 1784125359 ; total 66 ; last 66) - unlinked 20000 (time 1784125400 ; total 107 ; last 41) - unlinked 30000 (time 1784125448 ; total 155 ; last 48) - unlinked 40000 (time 1784125491 ; total 198 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 246 seconds: 201.752029 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local oleg321-server: modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2