== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:31:33 (1789133493) start mds service on oleg144-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg144-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1789133571.74 total 10.01 last 20.18) - open/close 317 (time 1789133581.75 total 20.02 last 11.49) - open/close 459 (time 1789133591.77 total 30.04 last 14.17) - open/close 569 (time 1789133602.11 total 40.38 last 10.64) - open/close 755 (time 1789133613.08 total 51.35 last 16.95) - open/close 926 (time 1789133623.11 total 61.38 last 17.06) - open/close 1081 (time 1789133633.59 total 71.86 last 14.79) - open/close 1275 (time 1789133643.61 total 81.88 last 19.36) - open/close 1418 (time 1789133654.00 total 92.27 last 13.76) - open/close 1560 (time 1789133664.20 total 102.47 last 13.92) - open/close 1808 (time 1789133674.22 total 112.49 last 24.76) - open/close 2060 (time 1789133684.69 total 122.95 last 24.08) - open/close 2336 (time 1789133695.61 total 133.88 last 25.26) - open/close 2568 (time 1789133705.61 total 143.88 last 23.20) - open/close 2829 (time 1789133716.22 total 154.49 last 24.60) - open/close 2945 (time 1789133727.31 total 165.58 last 10.46) - open/close 3134 (time 1789133739.63 total 177.90 last 15.34) - open/close 3358 (time 1789133750.00 total 188.27 last 21.60) - open/close 3503 (time 1789133760.38 total 198.65 last 13.97) - open/close 3606 (time 1789133770.43 total 208.70 last 10.25) - open/close 3887 (time 1789133780.44 total 218.70 last 28.08) - open/close 4156 (time 1789133790.47 total 228.73 last 26.82) - open/close 4366 (time 1789133801.28 total 239.55 last 19.42) - open/close 4601 (time 1789133811.28 total 249.55 last 23.49) - open/close 4743 (time 1789133821.80 total 260.07 last 13.50) - open/close 4927 (time 1789133831.81 total 270.08 last 18.38) - open/close 5159 (time 1789133841.91 total 280.18 last 22.97) - open/close 5421 (time 1789133851.93 total 290.19 last 26.17) - open/close 5648 (time 1789133862.35 total 300.62 last 21.78) - open/close 5829 (time 1789133873.37 total 311.63 last 16.43) - open/close 6041 (time 1789133883.37 total 321.63 last 21.20) - open/close 6236 (time 1789133893.44 total 331.71 last 19.36) - open/close 6390 (time 1789133903.45 total 341.72 last 15.38) - open/close 6626 (time 1789133914.24 total 352.50 last 21.89) - open/close 6742 (time 1789133924.35 total 362.62 last 11.47) - open/close 6803 (time 1789133934.36 total 372.62 last 6.10) - open/close 6939 (time 1789133944.77 total 383.03 last 13.07) - open/close 7198 (time 1789133954.77 total 393.03 last 25.90) - open/close 7380 (time 1789133966.62 total 404.89 last 15.36) - open/close 7577 (time 1789133976.62 total 414.89 last 19.69) - open/close 7754 (time 1789133986.68 total 424.95 last 17.60) - open/close 7851 (time 1789133996.94 total 435.20 last 9.46) - open/close 7990 (time 1789134006.96 total 445.23 last 13.86) - open/close 8094 (time 1789134017.01 total 455.27 last 10.36) - open/close 8296 (time 1789134027.59 total 465.86 last 19.08) - open/close 8470 (time 1789134037.73 total 475.99 last 17.17) - open/close 8618 (time 1789134048.08 total 486.35 last 14.29) - open/close 8828 (time 1789134059.16 total 497.43 last 18.96) - open/close 9058 (time 1789134069.17 total 507.43 last 22.98) - open/close 9209 (time 1789134080.19 total 518.46 last 13.69) - open/close 9342 (time 1789134090.56 total 528.83 last 12.83) - open/close 9549 (time 1789134101.39 total 539.66 last 19.11) - open/close 9807 (time 1789134111.93 total 550.20 last 24.48) - open/close 10000 (time 1789134119.12 total 557.38 last 26.87) - open/close 10238 (time 1789134129.13 total 567.39 last 23.77) - open/close 10478 (time 1789134139.15 total 577.41 last 23.95) - open/close 10706 (time 1789134149.72 total 587.98 last 21.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10897) error: No space left on device total: 10897 open/close in 596.54 seconds: 18.27 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76104) - No such process - unlinked 0 (time 1789134173 ; total 0 ; last 0) - unlinked 10000 (time 1789134256 ; total 83 ; last 83) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-10897) error: No such file or directory total: 10897 unlinks in 90 seconds: 121.077774 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2