== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:00:56 (1781236856) start mds service on oleg647-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg647-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg647-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg647-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1781236914.51 total 10.77 last 20.89) - open/close 456 (time 1781236925.39 total 21.65 last 21.23) - open/close 633 (time 1781236935.63 total 31.89 last 17.29) - open/close 890 (time 1781236946.39 total 42.65 last 23.88) - open/close 1150 (time 1781236956.88 total 53.14 last 24.79) - open/close 1591 (time 1781236967.25 total 63.50 last 42.53) - open/close 1808 (time 1781236977.68 total 73.93 last 20.81) - open/close 1950 (time 1781236987.73 total 83.98 last 14.13) - open/close 2087 (time 1781236997.73 total 93.98 last 13.70) - open/close 2474 (time 1781237007.73 total 103.99 last 38.70) - open/close 2611 (time 1781237018.00 total 114.26 last 13.34) - open/close 2816 (time 1781237028.82 total 125.07 last 18.95) - open/close 3088 (time 1781237039.31 total 135.57 last 25.91) - open/close 3284 (time 1781237049.87 total 146.13 last 18.57) - open/close 3547 (time 1781237059.87 total 156.13 last 26.29) - open/close 3802 (time 1781237070.09 total 166.35 last 24.95) - open/close 4040 (time 1781237080.10 total 176.35 last 23.79) - open/close 4329 (time 1781237090.33 total 186.58 last 28.25) - open/close 4597 (time 1781237100.33 total 196.59 last 26.80) - open/close 4820 (time 1781237110.99 total 207.25 last 20.92) - open/close 5180 (time 1781237120.99 total 217.25 last 35.99) - open/close 5450 (time 1781237131.85 total 228.11 last 24.86) - open/close 5697 (time 1781237142.80 total 239.05 last 22.57) - open/close 5945 (time 1781237153.04 total 249.30 last 24.21) - open/close 6217 (time 1781237163.60 total 259.86 last 25.76) - open/close 6530 (time 1781237174.48 total 270.74 last 28.76) - open/close 6668 (time 1781237184.65 total 280.91 last 13.57) - open/close 6850 (time 1781237195.09 total 291.35 last 17.43) - open/close 7104 (time 1781237205.10 total 301.35 last 25.39) - open/close 7376 (time 1781237215.82 total 312.08 last 25.36) - open/close 7864 (time 1781237226.13 total 322.39 last 47.35) - open/close 8142 (time 1781237236.25 total 332.50 last 27.48) - open/close 8394 (time 1781237247.25 total 343.50 last 22.91) - open/close 8554 (time 1781237257.68 total 353.93 last 15.34) - open/close 8731 (time 1781237268.05 total 364.31 last 17.07) - open/close 9018 (time 1781237278.05 total 374.31 last 28.70) - open/close 9266 (time 1781237288.05 total 384.31 last 24.79) - open/close 9900 (time 1781237298.06 total 394.31 last 63.39) - open/close 12332 (time 1781237308.06 total 404.31 last 243.17) - open/close 13934 (time 1781237318.06 total 414.32 last 160.16) - open/close 15848 (time 1781237328.06 total 424.32 last 191.33) - open/close 17393 (time 1781237338.07 total 434.32 last 154.43) - open/close 19793 (time 1781237348.07 total 444.33 last 239.96) - open/close 22148 (time 1781237358.07 total 454.33 last 235.42) - open/close 23814 (time 1781237368.08 total 464.34 last 166.43) - open/close 25338 (time 1781237378.09 total 474.35 last 152.31) - open/close 26713 (time 1781237388.10 total 484.35 last 137.41) - open/close 28613 (time 1781237398.10 total 494.35 last 189.96) - open/close 30000 (time 1781237405.27 total 501.52 last 193.44) - open/close 31483 (time 1781237415.27 total 511.53 last 148.23) - open/close 33429 (time 1781237425.27 total 521.53 last 194.58) - open/close 35357 (time 1781237435.28 total 531.53 last 192.77) - open/close 36845 (time 1781237445.28 total 541.53 last 148.77) - open/close 38198 (time 1781237455.28 total 551.54 last 135.22) - open/close 39675 (time 1781237465.28 total 561.54 last 147.67) - open/close 40000 (time 1781237467.83 total 564.09 last 127.54) - open/close 41203 (time 1781237477.84 total 574.09 last 120.27) - open/close 42514 (time 1781237487.86 total 584.11 last 130.84) - open/close 43808 (time 1781237497.86 total 594.12 last 129.35) - open/close 45266 (time 1781237507.86 total 604.12 last 145.77) - open/close 46991 (time 1781237517.87 total 614.12 last 172.40) - open/close 48589 (time 1781237527.87 total 624.13 last 159.73) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 632.30 seconds: 78.49 ops/second - unlinked 0 (time 1781237538 ; total 0 ; last 0) - unlinked 10000 (time 1781237575 ; total 37 ; last 37) - unlinked 20000 (time 1781237602 ; total 64 ; last 27) - unlinked 30000 (time 1781237632 ; total 94 ; last 30) - unlinked 40000 (time 1781237660 ; total 122 ; last 28) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 153 seconds: 324.385620 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg647-server unloading modules via unload_modules_local on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing unload_modules_local oleg647-server: modules unloaded. pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2