== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:58:46 (1780387126) start mds service on oleg104-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg104-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg104-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1780387182.30 total 10.43 last 22.43) - open/close 522 (time 1780387193.30 total 21.44 last 26.17) - open/close 726 (time 1780387203.73 total 31.87 last 19.56) - open/close 1317 (time 1780387213.78 total 41.92 last 58.82) - open/close 1603 (time 1780387226.07 total 54.21 last 23.27) - open/close 2025 (time 1780387237.01 total 65.15 last 38.56) - open/close 2451 (time 1780387248.02 total 76.16 last 38.69) - open/close 2817 (time 1780387258.78 total 86.91 last 34.03) - open/close 3305 (time 1780387269.78 total 97.92 last 44.35) - open/close 3563 (time 1780387280.02 total 108.16 last 25.19) - open/close 4023 (time 1780387290.20 total 118.34 last 45.18) - open/close 4396 (time 1780387301.21 total 129.35 last 33.89) - open/close 4660 (time 1780387312.16 total 140.29 last 24.12) - open/close 4863 (time 1780387322.58 total 150.72 last 19.48) - open/close 5315 (time 1780387333.53 total 161.66 last 41.28) - open/close 5729 (time 1780387344.15 total 172.29 last 38.97) - open/close 5871 (time 1780387354.20 total 182.33 last 14.14) - open/close 6286 (time 1780387364.63 total 192.77 last 39.78) - open/close 6518 (time 1780387375.45 total 203.59 last 21.44) - open/close 6761 (time 1780387385.45 total 213.59 last 24.29) - open/close 7017 (time 1780387396.25 total 224.38 last 23.72) - open/close 7233 (time 1780387406.55 total 234.68 last 20.97) - open/close 7860 (time 1780387417.11 total 245.24 last 59.38) - open/close 8113 (time 1780387427.54 total 255.68 last 24.25) - open/close 8598 (time 1780387437.91 total 266.05 last 46.76) - open/close 8925 (time 1780387448.85 total 276.99 last 29.90) - open/close 9255 (time 1780387459.54 total 287.68 last 30.87) - open/close 9538 (time 1780387469.72 total 297.85 last 27.81) - open/close 9732 (time 1780387479.96 total 308.10 last 18.94) - open/close 9970 (time 1780387490.13 total 318.27 last 23.40) - open/close 10546 (time 1780387500.14 total 328.27 last 57.57) - open/close 11238 (time 1780387510.61 total 338.75 last 66.04) - open/close 11848 (time 1780387520.91 total 349.05 last 59.21) - open/close 12466 (time 1780387531.16 total 359.29 last 60.34) - open/close 15840 (time 1780387541.16 total 369.30 last 337.34) - open/close 20000 (time 1780387550.68 total 378.82 last 436.94) - open/close 24484 (time 1780387560.68 total 388.82 last 448.34) - open/close 28661 (time 1780387570.68 total 398.82 last 417.70) - open/close 30000 (time 1780387574.22 total 402.36 last 378.16) - open/close 33780 (time 1780387584.22 total 412.36 last 377.95) - open/close 37645 (time 1780387594.22 total 422.36 last 386.50) - open/close 40000 (time 1780387600.26 total 428.40 last 390.03) - open/close 44279 (time 1780387610.26 total 438.40 last 427.78) - open/close 47772 (time 1780387620.26 total 448.40 last 349.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 452.86 seconds: 109.60 ops/second - unlinked 0 (time 1780387626 ; total 0 ; last 0) - unlinked 10000 (time 1780387641 ; total 15 ; last 15) - unlinked 20000 (time 1780387656 ; total 30 ; last 15) - unlinked 30000 (time 1780387670 ; total 44 ; last 14) - unlinked 40000 (time 1780387687 ; total 61 ; last 17) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 77 seconds: 644.558472 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2