== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:08:43 (1783973323) start mds service on oleg647-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg647-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg647-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg647-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 223 (time 1783973434.93 total 10.02 last 22.25) - open/close 460 (time 1783973445.33 total 20.43 last 22.78) - open/close 715 (time 1783973455.34 total 30.44 last 25.48) - open/close 906 (time 1783973465.75 total 40.84 last 18.35) - open/close 1142 (time 1783973475.93 total 51.02 last 23.19) - open/close 1428 (time 1783973486.10 total 61.20 last 28.10) - open/close 1686 (time 1783973496.36 total 71.46 last 25.16) - open/close 1929 (time 1783973506.71 total 81.80 last 23.49) - open/close 2178 (time 1783973516.72 total 91.82 last 24.85) - open/close 2373 (time 1783973526.73 total 101.82 last 19.50) - open/close 2607 (time 1783973536.91 total 112.01 last 22.97) - open/close 2853 (time 1783973546.96 total 122.06 last 24.49) - open/close 3081 (time 1783973557.58 total 132.68 last 21.47) - open/close 3307 (time 1783973568.46 total 143.56 last 20.77) - open/close 3501 (time 1783973579.03 total 154.13 last 18.36) - open/close 3736 (time 1783973589.04 total 164.13 last 23.48) - open/close 3958 (time 1783973599.06 total 174.16 last 22.15) - open/close 4177 (time 1783973609.32 total 184.41 last 21.35) - open/close 4382 (time 1783973619.48 total 194.58 last 20.17) - open/close 4608 (time 1783973630.16 total 205.26 last 21.16) - open/close 4840 (time 1783973640.22 total 215.32 last 23.06) - open/close 5017 (time 1783973650.26 total 225.35 last 17.64) - open/close 5240 (time 1783973660.26 total 235.36 last 22.28) - open/close 5503 (time 1783973670.95 total 246.04 last 24.62) - open/close 5760 (time 1783973680.95 total 256.05 last 25.69) - open/close 5962 (time 1783973691.02 total 266.12 last 20.05) - open/close 6226 (time 1783973701.04 total 276.13 last 26.37) - open/close 6453 (time 1783973711.04 total 286.13 last 22.70) - open/close 6634 (time 1783973721.30 total 296.39 last 17.64) - open/close 6708 (time 1783973731.75 total 306.85 last 7.08) - open/close 6852 (time 1783973742.80 total 317.90 last 13.03) - open/close 7069 (time 1783973753.43 total 328.53 last 20.42) - open/close 7310 (time 1783973763.86 total 338.96 last 23.11) - open/close 7414 (time 1783973775.15 total 350.24 last 9.21) - open/close 7631 (time 1783973785.19 total 360.28 last 21.61) - open/close 7821 (time 1783973795.19 total 370.29 last 19.00) - open/close 8064 (time 1783973805.98 total 381.08 last 22.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8273) error: No space left on device total: 8273 open/close in 389.15 seconds: 21.26 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114722) - No such process - unlinked 0 (time 1783973823 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8273) error: No such file or directory total: 8273 unlinks in 62 seconds: 133.435486 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg647-server unloading modules via unload_modules_local on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing unload_modules_local oleg647-server: modules unloaded. pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2