== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:00:52 (1787608852) start mds service on oleg154-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing load_modules_local oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg154-server: Loading modules from /home/green/git/lustre-release/lustre oleg154-server: detected 4 online CPUs by sysfs oleg154-server: MODOPTS_LIBCFS= oleg154-server: Force libcfs to create 2 CPU partitions oleg154-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg154-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg154-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg154-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1 Started lustre-OST0000 oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg154-client.virtnet: -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 117 (time 1787608939.15 total 10.18 last 11.49) - open/close 329 (time 1787608949.90 total 20.93 last 19.73) - open/close 466 (time 1787608960.97 total 32.00 last 12.38) - open/close 705 (time 1787608971.79 total 42.82 last 22.08) - open/close 953 (time 1787608982.79 total 53.82 last 22.56) - open/close 1204 (time 1787608995.10 total 66.13 last 20.39) - open/close 1351 (time 1787609005.11 total 76.14 last 14.68) - open/close 1461 (time 1787609015.44 total 86.47 last 10.65) - open/close 1695 (time 1787609025.48 total 96.50 last 23.32) - open/close 1854 (time 1787609035.52 total 106.54 last 15.84) - open/close 2081 (time 1787609046.05 total 117.08 last 21.55) - open/close 2335 (time 1787609056.34 total 127.37 last 24.69) - open/close 2493 (time 1787609067.34 total 138.37 last 14.36) - open/close 2596 (time 1787609078.22 total 149.24 last 9.47) - open/close 2719 (time 1787609088.25 total 159.28 last 12.26) - open/close 2848 (time 1787609098.80 total 169.83 last 12.22) - open/close 3034 (time 1787609108.82 total 179.84 last 18.58) - open/close 3218 (time 1787609118.82 total 189.84 last 18.40) - open/close 3396 (time 1787609128.88 total 199.90 last 17.69) - open/close 3483 (time 1787609139.52 total 210.55 last 8.17) - open/close 3558 (time 1787609149.54 total 220.57 last 7.49) - open/close 3784 (time 1787609159.56 total 230.59 last 22.55) - open/close 4061 (time 1787609170.57 total 241.60 last 25.16) - open/close 4267 (time 1787609180.58 total 251.61 last 20.59) - open/close 4516 (time 1787609190.58 total 261.61 last 24.90) - open/close 4707 (time 1787609201.21 total 272.24 last 17.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-4817) error: No space left on device total: 4817 open/close in 278.50 seconds: 17.30 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76058) - No such process - unlinked 0 (time 1787609212 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-4817) error: No such file or directory total: 4817 unlinks in 33 seconds: 145.969696 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg154-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg154-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server stop mds service on oleg154-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server unloading modules via unload_modules_local on: 'oleg154-server' oleg154-server: oleg154-server.virtnet: executing unload_modules_local oleg154-server: modules unloaded. pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2 pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2