== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:24:03 (1782534243) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg255-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg255-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg255-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 500 (time 1782534299.47 total 10.45 last 47.84) - open/close 996 (time 1782534309.78 total 20.75 last 48.14) - open/close 1545 (time 1782534320.34 total 31.32 last 51.98) - open/close 1926 (time 1782534330.45 total 41.43 last 37.68) - open/close 2575 (time 1782534341.33 total 52.31 last 59.65) - open/close 3234 (time 1782534352.27 total 63.25 last 60.22) - open/close 3845 (time 1782534362.96 total 73.94 last 57.17) - open/close 4309 (time 1782534373.20 total 84.18 last 45.31) - open/close 4740 (time 1782534383.57 total 94.55 last 41.58) - open/close 5138 (time 1782534393.81 total 104.79 last 38.86) - open/close 5541 (time 1782534403.99 total 114.96 last 39.61) - open/close 6240 (time 1782534414.04 total 125.01 last 69.56) - open/close 6848 (time 1782534424.79 total 135.77 last 56.52) - open/close 7478 (time 1782534434.83 total 145.81 last 62.74) - open/close 8083 (time 1782534445.59 total 156.56 last 56.26) - open/close 8773 (time 1782534456.47 total 167.44 last 63.41) - open/close 9375 (time 1782534467.35 total 178.32 last 55.34) - open/close 9868 (time 1782534477.40 total 188.37 last 49.06) - open/close 10547 (time 1782534488.41 total 199.38 last 61.67) - open/close 11210 (time 1782534498.41 total 209.38 last 66.30) - open/close 11657 (time 1782534508.76 total 219.73 last 43.18) - open/close 12027 (time 1782534518.93 total 229.91 last 36.37) - open/close 12582 (time 1782534529.62 total 240.60 last 51.91) - open/close 13021 (time 1782534539.99 total 250.97 last 42.34) - open/close 13590 (time 1782534550.74 total 261.72 last 52.92) - open/close 14201 (time 1782534561.49 total 272.47 last 56.82) - open/close 14827 (time 1782534572.31 total 283.28 last 57.89) - open/close 15389 (time 1782534582.87 total 293.85 last 53.20) - open/close 18085 (time 1782534592.87 total 303.85 last 269.58) - open/close 20000 (time 1782534598.57 total 309.54 last 336.23) - open/close 24195 (time 1782534608.57 total 319.54 last 419.39) - open/close 28672 (time 1782534618.57 total 329.55 last 447.63) - open/close 30000 (time 1782534621.33 total 332.30 last 481.60) - open/close 34573 (time 1782534631.33 total 342.30 last 457.24) - open/close 39168 (time 1782534641.33 total 352.31 last 459.44) - open/close 43729 (time 1782534651.33 total 362.31 last 456.10) - open/close 48206 (time 1782534661.33 total 372.31 last 447.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 375.71 seconds: 132.10 ops/second - unlinked 0 (time 1782534666 ; total 0 ; last 0) - unlinked 10000 (time 1782534679 ; total 13 ; last 13) - unlinked 20000 (time 1782534692 ; total 26 ; last 13) - unlinked 30000 (time 1782534703 ; total 37 ; last 11) - unlinked 40000 (time 1782534714 ; total 48 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 60 seconds: 827.183350 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2