== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:55:13 (1781625313) start mds service on oleg350-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions oleg350-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg350-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg350-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 286 (time 1781625381.49 total 10.48 last 27.30) - open/close 568 (time 1781625392.38 total 21.36 last 25.91) - open/close 919 (time 1781625402.69 total 31.67 last 34.05) - open/close 1167 (time 1781625413.06 total 42.04 last 23.91) - open/close 1403 (time 1781625423.42 total 52.40 last 22.77) - open/close 1702 (time 1781625433.43 total 62.41 last 29.87) - open/close 1932 (time 1781625443.59 total 72.57 last 22.65) - open/close 2230 (time 1781625456.50 total 85.49 last 23.07) - open/close 2560 (time 1781625466.50 total 95.49 last 33.00) - open/close 2861 (time 1781625477.25 total 106.23 last 28.02) - open/close 3217 (time 1781625487.25 total 116.24 last 35.58) - open/close 3445 (time 1781625497.53 total 126.51 last 22.19) - open/close 3759 (time 1781625508.42 total 137.40 last 28.84) - open/close 4108 (time 1781625518.52 total 147.50 last 34.54) - open/close 4443 (time 1781625529.47 total 158.45 last 30.61) - open/close 4767 (time 1781625540.41 total 169.39 last 29.60) - open/close 5082 (time 1781625550.97 total 179.96 last 29.82) - open/close 5406 (time 1781625561.72 total 190.70 last 30.15) - open/close 5706 (time 1781625572.28 total 201.26 last 28.41) - open/close 6037 (time 1781625583.23 total 212.21 last 30.23) - open/close 6324 (time 1781625593.66 total 222.64 last 27.53) - open/close 6652 (time 1781625604.49 total 233.47 last 30.28) - open/close 6929 (time 1781625615.10 total 244.08 last 26.10) - open/close 7255 (time 1781625626.04 total 255.02 last 29.80) - open/close 7608 (time 1781625636.23 total 265.21 last 34.65) - open/close 7940 (time 1781625647.10 total 276.08 last 30.55) - open/close 8253 (time 1781625657.85 total 286.83 last 29.11) - open/close 8513 (time 1781625668.86 total 297.84 last 23.62) - open/close 8807 (time 1781625679.16 total 308.14 last 28.53) - open/close 9033 (time 1781625690.10 total 319.09 last 20.65) - open/close 9152 (time 1781625700.10 total 329.09 last 11.90) - open/close 9400 (time 1781625710.10 total 339.09 last 24.80) - open/close 9703 (time 1781625720.57 total 349.55 last 28.94) - open/close 9955 (time 1781625731.32 total 360.30 last 23.45) - open/close 10126 (time 1781625741.75 total 370.73 last 16.39) - open/close 10223 (time 1781625752.57 total 381.55 last 8.97) - open/close 10416 (time 1781625762.81 total 391.79 last 18.85) - open/close 10660 (time 1781625773.11 total 402.10 last 23.67) - open/close 10767 (time 1781625784.12 total 413.11 last 9.72) - open/close 10928 (time 1781625794.23 total 423.21 last 15.93) - open/close 11143 (time 1781625804.92 total 433.90 last 20.11) - open/close 12630 (time 1781625814.93 total 443.91 last 148.64) - open/close 14759 (time 1781625824.93 total 453.91 last 212.86) - open/close 17140 (time 1781625834.93 total 463.91 last 238.06) - open/close 19477 (time 1781625844.93 total 473.91 last 233.68) - open/close 20000 (time 1781625847.15 total 476.14 last 235.19) - open/close 22049 (time 1781625857.16 total 486.14 last 204.84) - open/close 23998 (time 1781625867.16 total 496.14 last 194.88) - open/close 26026 (time 1781625877.16 total 506.14 last 202.79) - open/close 28231 (time 1781625887.16 total 516.14 last 220.50) - open/close 30000 (time 1781625896.01 total 524.99 last 199.84) - open/close 32403 (time 1781625906.01 total 534.99 last 240.27) - open/close 34798 (time 1781625916.01 total 544.99 last 239.49) - open/close 37091 (time 1781625926.02 total 555.00 last 229.20) - open/close 39128 (time 1781625936.02 total 565.00 last 203.70) - open/close 40000 (time 1781625939.23 total 568.22 last 271.00) - open/close 42613 (time 1781625949.23 total 578.22 last 261.30) - open/close 45373 (time 1781625959.24 total 588.22 last 275.93) - open/close 47867 (time 1781625969.24 total 598.22 last 249.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 604.77 seconds: 82.07 ops/second - unlinked 0 (time 1781625977 ; total 0 ; last 0) - unlinked 10000 (time 1781625999 ; total 22 ; last 22) - unlinked 20000 (time 1781626024 ; total 47 ; last 25) - unlinked 30000 (time 1781626049 ; total 72 ; last 25) - unlinked 40000 (time 1781626070 ; total 93 ; last 21) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 112 seconds: 443.133942 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2