== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:26:46 (1786354006) start mds service on oleg347-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg347-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 242 (time 1786354051.53 total 10.00 last 24.19) - open/close 567 (time 1786354062.29 total 20.76 last 30.21) - open/close 878 (time 1786354072.91 total 31.38 last 29.29) - open/close 1182 (time 1786354083.47 total 41.94 last 28.79) - open/close 1477 (time 1786354093.90 total 52.38 last 28.26) - open/close 1722 (time 1786354103.91 total 62.38 last 24.50) - open/close 1913 (time 1786354114.50 total 72.98 last 18.02) - open/close 2079 (time 1786354124.94 total 83.41 last 15.91) - open/close 2220 (time 1786354135.18 total 93.65 last 13.77) - open/close 2448 (time 1786354145.18 total 103.65 last 22.79) - open/close 2683 (time 1786354155.21 total 113.68 last 23.44) - open/close 3000 (time 1786354165.83 total 124.31 last 29.83) - open/close 3281 (time 1786354176.20 total 134.67 last 27.10) - open/close 3563 (time 1786354186.64 total 145.11 last 27.03) - open/close 3893 (time 1786354197.26 total 155.73 last 31.06) - open/close 4193 (time 1786354207.76 total 166.23 last 28.59) - open/close 4396 (time 1786354218.51 total 176.98 last 18.88) - open/close 4601 (time 1786354229.19 total 187.67 last 19.19) - open/close 4856 (time 1786354239.25 total 197.72 last 25.37) - open/close 5157 (time 1786354249.35 total 207.82 last 29.78) - open/close 5419 (time 1786354259.36 total 217.83 last 26.19) - open/close 5696 (time 1786354269.64 total 228.11 last 26.94) - open/close 5936 (time 1786354279.70 total 238.17 last 23.87) - open/close 6200 (time 1786354289.87 total 248.34 last 25.95) - open/close 6510 (time 1786354299.87 total 258.34 last 31.00) - open/close 6752 (time 1786354309.88 total 268.35 last 24.18) - open/close 6986 (time 1786354320.78 total 279.25 last 21.47) - open/close 7242 (time 1786354331.40 total 289.87 last 24.10) - open/close 7383 (time 1786354341.71 total 300.18 last 13.68) - open/close 7646 (time 1786354351.75 total 310.22 last 26.18) - open/close 7851 (time 1786354362.51 total 320.98 last 19.06) - open/close 8063 (time 1786354373.26 total 331.73 last 19.72) - open/close 8302 (time 1786354384.01 total 342.48 last 22.22) - open/close 8531 (time 1786354394.76 total 353.23 last 21.30) - open/close 8712 (time 1786354405.33 total 363.80 last 17.13) - open/close 9004 (time 1786354415.75 total 374.22 last 28.00) - open/close 9220 (time 1786354426.44 total 384.91 last 20.21) - open/close 9396 (time 1786354436.69 total 395.16 last 17.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9457) error: No space left on device total: 9457 open/close in 398.62 seconds: 23.72 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (75902) - No such process - unlinked 0 (time 1786354444 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9457) error: No such file or directory total: 9457 unlinks in 28 seconds: 337.750000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2