== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:59:07 (1786193947) start mds service on oleg233-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg233-server: detected 4 online CPUs by sysfs oleg233-server: MODOPTS_LIBCFS= oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg233-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg233-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg233-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 775 (time 1786194000.95 total 11.03 last 70.24) - open/close 1325 (time 1786194011.57 total 21.66 last 51.77) - open/close 1802 (time 1786194022.20 total 32.28 last 44.89) - open/close 2378 (time 1786194032.88 total 42.97 last 53.90) - open/close 3096 (time 1786194043.06 total 53.15 last 70.56) - open/close 3882 (time 1786194053.43 total 63.51 last 75.81) - open/close 4539 (time 1786194064.18 total 74.27 last 61.10) - open/close 5075 (time 1786194074.68 total 84.76 last 51.08) - open/close 5735 (time 1786194085.68 total 95.77 last 59.96) - open/close 6331 (time 1786194096.44 total 106.52 last 55.42) - open/close 6950 (time 1786194106.44 total 116.52 last 61.89) - open/close 7592 (time 1786194116.53 total 126.62 last 63.61) - open/close 8236 (time 1786194127.35 total 137.44 last 59.53) - open/close 8920 (time 1786194138.29 total 148.38 last 62.51) - open/close 9569 (time 1786194149.11 total 159.20 last 59.99) - open/close 10000 (time 1786194156.30 total 166.38 last 59.96) - open/close 10618 (time 1786194166.97 total 177.05 last 57.93) - open/close 11187 (time 1786194177.40 total 187.48 last 54.56) - open/close 11721 (time 1786194187.76 total 197.85 last 51.50) - open/close 12396 (time 1786194198.52 total 208.60 last 62.76) - open/close 13181 (time 1786194208.76 total 218.85 last 76.65) - open/close 13981 (time 1786194219.12 total 229.21 last 77.19) - open/close 14822 (time 1786194229.43 total 239.51 last 81.62) - open/close 15595 (time 1786194239.60 total 249.69 last 75.97) - open/close 16342 (time 1786194250.55 total 260.64 last 68.25) - open/close 17053 (time 1786194260.60 total 270.68 last 70.76) - open/close 17802 (time 1786194270.64 total 280.73 last 74.55) - open/close 18575 (time 1786194280.89 total 290.98 last 75.45) - open/close 19332 (time 1786194291.00 total 301.09 last 74.87) - open/close 20000 (time 1786194297.64 total 307.73 last 100.53) - open/close 24524 (time 1786194307.65 total 317.73 last 452.21) - open/close 28886 (time 1786194317.65 total 327.74 last 436.15) - open/close 30000 (time 1786194320.06 total 330.15 last 461.98) - open/close 34620 (time 1786194330.06 total 340.15 last 461.90) - open/close 39158 (time 1786194340.06 total 350.15 last 453.73) - open/close 43833 (time 1786194350.07 total 360.15 last 467.47) - open/close 48458 (time 1786194360.07 total 370.15 last 462.41) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 372.74 seconds: 133.15 ops/second - unlinked 0 (time 1786194364 ; total 0 ; last 0) - unlinked 10000 (time 1786194375 ; total 11 ; last 11) - unlinked 20000 (time 1786194387 ; total 23 ; last 12) - unlinked 30000 (time 1786194398 ; total 34 ; last 11) - unlinked 40000 (time 1786194411 ; total 47 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 59 seconds: 841.203369 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg233-server unloading modules via unload_modules_local on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: modules unloaded. pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2