== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:11:54 (1773105114) start mds service on oleg147-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg147-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg147-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg147-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg147-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 342 (time 1773105172.63 total 10.70 last 31.98) - open/close 600 (time 1773105182.93 total 21.00 last 25.04) - open/close 964 (time 1773105195.22 total 33.29 last 29.62) - open/close 1185 (time 1773105205.22 total 43.29 last 22.10) - open/close 1347 (time 1773105215.22 total 53.29 last 16.20) - open/close 1742 (time 1773105225.81 total 63.88 last 37.29) - open/close 2417 (time 1773105236.11 total 74.18 last 65.52) - open/close 2783 (time 1773105246.74 total 84.81 last 34.45) - open/close 2963 (time 1773105257.75 total 95.81 last 16.35) - open/close 3194 (time 1773105267.92 total 105.99 last 22.70) - open/close 3389 (time 1773105278.81 total 116.88 last 17.91) - open/close 3779 (time 1773105289.17 total 127.24 last 37.64) - open/close 4740 (time 1773105300.12 total 138.19 last 87.76) - open/close 5552 (time 1773105310.67 total 148.74 last 76.93) - open/close 6108 (time 1773105323.54 total 161.61 last 43.22) - open/close 6559 (time 1773105334.16 total 172.23 last 42.45) - open/close 7180 (time 1773105345.17 total 183.24 last 56.43) - open/close 7686 (time 1773105355.80 total 193.86 last 47.62) - open/close 8363 (time 1773105366.80 total 204.87 last 61.49) - open/close 9013 (time 1773105376.81 total 214.88 last 64.96) - open/close 9851 (time 1773105387.16 total 225.22 last 81.02) - open/close 10575 (time 1773105397.46 total 235.53 last 70.26) - open/close 11421 (time 1773105408.15 total 246.22 last 79.15) - open/close 11887 (time 1773105419.09 total 257.16 last 42.58) - open/close 12282 (time 1773105431.57 total 269.64 last 31.65) - open/close 12770 (time 1773105442.32 total 280.39 last 45.40) - open/close 13374 (time 1773105452.32 total 290.39 last 60.39) - open/close 14158 (time 1773105462.48 total 300.55 last 77.18) - open/close 14406 (time 1773105473.49 total 311.56 last 22.53) - open/close 15021 (time 1773105483.49 total 321.56 last 61.49) - open/close 15231 (time 1773105494.23 total 332.30 last 19.55) - open/close 15609 (time 1773105504.59 total 342.66 last 36.49) - open/close 18193 (time 1773105514.59 total 352.66 last 258.35) - open/close 20000 (time 1773105518.67 total 356.74 last 442.99) - open/close 24557 (time 1773105528.68 total 366.75 last 455.59) - open/close 28884 (time 1773105538.68 total 376.75 last 432.70) - open/close 30000 (time 1773105541.74 total 379.81 last 363.91) - open/close 33366 (time 1773105551.74 total 389.81 last 336.53) - open/close 36663 (time 1773105561.75 total 399.82 last 329.55) - open/close 39936 (time 1773105571.75 total 409.82 last 327.28) - open/close 43355 (time 1773105581.75 total 419.82 last 341.80) - open/close 46377 (time 1773105591.75 total 429.82 last 302.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 439.69 seconds: 112.88 ops/second - unlinked 0 (time 1773105603 ; total 0 ; last 0) - unlinked 10000 (time 1773105620 ; total 17 ; last 17) - unlinked 20000 (time 1773105637 ; total 34 ; last 17) - unlinked 30000 (time 1773105652 ; total 49 ; last 15) - unlinked 40000 (time 1773105673 ; total 70 ; last 21) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 95 seconds: 522.442078 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg147-server unloading modules via unload_modules_local on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing unload_modules_local oleg147-server: modules unloaded. pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2