== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:33:34 (1783074814) start mds service on oleg439-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg439-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg439-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg439-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 793 (time 1783074870.88 total 10.00 last 79.29) - open/close 1460 (time 1783074880.88 total 20.00 last 66.68) - open/close 2011 (time 1783074890.93 total 30.06 last 54.81) - open/close 2593 (time 1783074901.11 total 40.23 last 57.22) - open/close 3318 (time 1783074912.05 total 51.17 last 66.23) - open/close 3931 (time 1783074922.06 total 61.18 last 61.26) - open/close 4512 (time 1783074932.06 total 71.18 last 58.09) - open/close 5066 (time 1783074943.09 total 82.21 last 50.22) - open/close 5775 (time 1783074953.46 total 92.58 last 68.38) - open/close 6364 (time 1783074963.46 total 102.58 last 58.89) - open/close 6982 (time 1783074973.56 total 112.68 last 61.22) - open/close 7525 (time 1783074983.60 total 122.73 last 54.04) - open/close 8200 (time 1783074993.97 total 133.10 last 65.10) - open/close 8730 (time 1783075004.79 total 143.91 last 49.01) - open/close 9384 (time 1783075014.79 total 153.91 last 65.39) - open/close 9771 (time 1783075025.33 total 164.45 last 36.71) - open/close 10000 (time 1783075032.53 total 171.66 last 31.80) - open/close 10119 (time 1783075043.44 total 182.56 last 10.91) - open/close 10408 (time 1783075053.88 total 193.00 last 27.70) - open/close 10824 (time 1783075063.88 total 203.00 last 41.60) - open/close 11457 (time 1783075073.88 total 213.00 last 63.29) - open/close 12158 (time 1783075083.88 total 223.00 last 70.09) - open/close 12823 (time 1783075094.83 total 233.96 last 60.70) - open/close 13398 (time 1783075105.59 total 244.71 last 53.45) - open/close 14095 (time 1783075116.02 total 255.14 last 66.85) - open/close 14649 (time 1783075126.26 total 265.38 last 54.10) - open/close 15263 (time 1783075136.37 total 275.49 last 60.72) - open/close 15941 (time 1783075146.55 total 285.67 last 66.64) - open/close 16583 (time 1783075157.56 total 296.68 last 58.30) - open/close 17217 (time 1783075167.60 total 306.73 last 63.11) - open/close 17731 (time 1783075177.97 total 317.09 last 49.58) - open/close 18627 (time 1783075187.97 total 327.10 last 89.57) - open/close 20000 (time 1783075191.85 total 330.97 last 354.52) - open/close 23119 (time 1783075201.85 total 340.97 last 311.87) - open/close 26258 (time 1783075211.85 total 350.97 last 313.84) - open/close 29718 (time 1783075221.85 total 360.97 last 345.89) - open/close 32972 (time 1783075231.86 total 370.98 last 325.30) - open/close 36869 (time 1783075241.86 total 380.98 last 389.70) - open/close 40000 (time 1783075250.51 total 389.63 last 361.88) - open/close 43919 (time 1783075260.51 total 399.63 last 391.84) - open/close 47867 (time 1783075270.51 total 409.63 last 394.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 414.76 seconds: 119.66 ops/second - unlinked 0 (time 1783075277 ; total 0 ; last 0) - unlinked 10000 (time 1783075291 ; total 14 ; last 14) - unlinked 20000 (time 1783075303 ; total 26 ; last 12) - unlinked 30000 (time 1783075316 ; total 39 ; last 13) - unlinked 40000 (time 1783075330 ; total 53 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 65 seconds: 763.553833 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local oleg439-server: modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2