== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:35:44 (1787628944) start mds service on oleg302-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg302-server: detected 4 online CPUs by sysfs oleg302-server: MODOPTS_LIBCFS= oleg302-server: Force libcfs to create 2 CPU partitions oleg302-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg302-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg302-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg302-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg302-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock 192.168.203.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1787629052.76 total 10.46 last 16.16) - open/close 354 (time 1787629063.18 total 20.88 last 17.74) - open/close 633 (time 1787629074.13 total 31.83 last 25.49) - open/close 889 (time 1787629084.77 total 42.47 last 24.07) - open/close 1162 (time 1787629095.05 total 52.75 last 26.54) - open/close 1475 (time 1787629105.95 total 63.65 last 28.71) - open/close 1716 (time 1787629115.96 total 73.66 last 24.08) - open/close 1948 (time 1787629125.96 total 83.66 last 23.19) - open/close 2218 (time 1787629135.96 total 93.66 last 27.00) - open/close 2495 (time 1787629146.48 total 104.18 last 26.33) - open/close 2802 (time 1787629157.13 total 114.83 last 28.84) - open/close 3123 (time 1787629167.15 total 124.85 last 32.05) - open/close 3364 (time 1787629177.37 total 135.07 last 23.57) - open/close 3668 (time 1787629187.38 total 145.08 last 30.37) - open/close 3910 (time 1787629197.40 total 155.10 last 24.15) - open/close 4165 (time 1787629207.41 total 165.11 last 25.48) - open/close 4392 (time 1787629218.03 total 175.73 last 21.37) - open/close 4612 (time 1787629228.43 total 186.13 last 21.15) - open/close 4872 (time 1787629238.67 total 196.37 last 25.40) - open/close 5132 (time 1787629248.86 total 206.56 last 25.51) - open/close 5435 (time 1787629259.66 total 217.36 last 28.05) - open/close 5702 (time 1787629270.60 total 228.30 last 24.40) - open/close 5912 (time 1787629280.65 total 238.35 last 20.90) - open/close 6113 (time 1787629291.31 total 249.01 last 18.87) - open/close 6365 (time 1787629301.73 total 259.43 last 24.17) - open/close 6632 (time 1787629312.59 total 270.29 last 24.58) - open/close 6881 (time 1787629323.10 total 280.80 last 23.69) - open/close 7138 (time 1787629333.11 total 290.81 last 25.68) - open/close 7463 (time 1787629343.58 total 301.27 last 31.07) - open/close 7706 (time 1787629354.12 total 311.82 last 23.04) - open/close 7959 (time 1787629364.13 total 321.82 last 25.29) - open/close 8221 (time 1787629374.14 total 331.84 last 26.16) - open/close 8485 (time 1787629384.48 total 342.18 last 25.53) - open/close 8649 (time 1787629395.76 total 353.46 last 14.54) - open/close 8860 (time 1787629405.77 total 363.47 last 21.08) - open/close 9099 (time 1787629415.78 total 373.48 last 23.88) - open/close 9392 (time 1787629426.51 total 384.21 last 27.29) - open/close 9673 (time 1787629436.52 total 394.22 last 28.07) - open/close 9928 (time 1787629447.53 total 405.23 last 23.17) - open/close 10000 (time 1787629449.58 total 407.28 last 35.12) - open/close 10271 (time 1787629459.59 total 417.29 last 27.07) - open/close 10557 (time 1787629469.84 total 427.54 last 27.90) - open/close 10810 (time 1787629480.79 total 438.49 last 23.11) - open/close 11072 (time 1787629493.56 total 451.26 last 20.52) - open/close 11336 (time 1787629503.56 total 461.26 last 26.39) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11377) error: No space left on device total: 11377 open/close in 461.87 seconds: 24.63 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114750) - No such process - unlinked 0 (time 1787629510 ; total 0 ; last 0) - unlinked 10000 (time 1787629572 ; total 62 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-11377) error: No such file or directory total: 11377 unlinks in 70 seconds: 162.528564 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg302-server unloading modules via unload_modules_local on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: modules unloaded. pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2