== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:44:12 (1785318252) start mds service on oleg629-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg629-server' oleg629-server: oleg629-server.virtnet: executing load_modules_local oleg629-server: Loading modules from /home/green/git/lustre-release/lustre oleg629-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg629-server: detected 4 online CPUs by sysfs oleg629-server: MODOPTS_LIBCFS= oleg629-server: Force libcfs to create 2 CPU partitions oleg629-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg629-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg629-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg629-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg629-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg629-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg629-server: oleg629-server.virtnet: executing set_default_debug -1 all pdsh@oleg629-client: oleg629-server: ssh exited with exit code 1 Started lustre-OST0000 oleg629-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg629-client.virtnet: -o user_xattr,flock 192.168.206.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 826 (time 1785318307.73 total 10.36 last 79.71) - open/close 1495 (time 1785318318.55 total 21.18 last 61.85) - open/close 2302 (time 1785318328.55 total 31.18 last 80.69) - open/close 3027 (time 1785318339.54 total 42.17 last 65.97) - open/close 3796 (time 1785318349.59 total 52.22 last 76.54) - open/close 4524 (time 1785318359.64 total 62.27 last 72.43) - open/close 5140 (time 1785318369.75 total 72.38 last 60.91) - open/close 5781 (time 1785318380.56 total 83.19 last 59.29) - open/close 6546 (time 1785318391.57 total 94.21 last 69.48) - open/close 7049 (time 1785318401.87 total 104.51 last 48.83) - open/close 7632 (time 1785318412.31 total 114.94 last 55.88) - open/close 8139 (time 1785318422.74 total 125.37 last 48.58) - open/close 8974 (time 1785318432.98 total 135.61 last 81.54) - open/close 9708 (time 1785318442.98 total 145.62 last 73.39) - open/close 10000 (time 1785318447.69 total 150.32 last 62.05) - open/close 10792 (time 1785318458.71 total 161.34 last 71.89) - open/close 11533 (time 1785318468.71 total 171.34 last 74.10) - open/close 12180 (time 1785318479.50 total 182.14 last 59.91) - open/close 12918 (time 1785318489.51 total 192.14 last 73.80) - open/close 13542 (time 1785318500.18 total 202.81 last 58.46) - open/close 13910 (time 1785318510.29 total 212.92 last 36.40) - open/close 14496 (time 1785318520.85 total 223.48 last 55.49) - open/close 15303 (time 1785318530.96 total 233.60 last 79.81) - open/close 16073 (time 1785318541.01 total 243.64 last 76.63) - open/close 16735 (time 1785318551.89 total 254.52 last 60.84) - open/close 17437 (time 1785318561.94 total 264.57 last 69.87) - open/close 18022 (time 1785318572.56 total 275.20 last 55.05) - open/close 18679 (time 1785318583.25 total 285.88 last 61.49) - open/close 19188 (time 1785318593.69 total 296.32 last 48.76) - open/close 19937 (time 1785318603.69 total 306.32 last 74.89) - open/close 24682 (time 1785318613.69 total 316.32 last 474.42) - open/close 29089 (time 1785318623.69 total 326.32 last 440.67) - open/close 30000 (time 1785318625.89 total 328.53 last 413.90) - open/close 34531 (time 1785318635.89 total 338.53 last 453.03) - open/close 38975 (time 1785318645.89 total 348.53 last 444.39) - open/close 40000 (time 1785318648.14 total 350.77 last 456.10) - open/close 44313 (time 1785318658.14 total 360.78 last 431.24) - open/close 48983 (time 1785318668.14 total 370.78 last 466.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 372.28 seconds: 133.32 ops/second - unlinked 0 (time 1785318671 ; total 0 ; last 0) - unlinked 10000 (time 1785318684 ; total 13 ; last 13) - unlinked 20000 (time 1785318695 ; total 24 ; last 11) - unlinked 30000 (time 1785318707 ; total 36 ; last 12) - unlinked 40000 (time 1785318718 ; total 47 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 61 seconds: 813.622925 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg629-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg629-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg629-server stop mds service on oleg629-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg629-server stop mds service on oleg629-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg629-server unloading modules via unload_modules_local on: 'oleg629-server' oleg629-server: oleg629-server.virtnet: executing unload_modules_local oleg629-server: modules unloaded. pdsh@oleg629-client: oleg629-client: ssh exited with exit code 2 pdsh@oleg629-client: oleg629-server: ssh exited with exit code 2 pdsh@oleg629-client: oleg629-client: ssh exited with exit code 2 pdsh@oleg629-client: oleg629-server: ssh exited with exit code 2