== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:49:00 (1786632540) start mds service on oleg117-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg117-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg117-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 93 (time 1786632670.80 total 10.02 last 9.28) - open/close 292 (time 1786632681.55 total 20.77 last 18.51) - open/close 396 (time 1786632692.17 total 31.39 last 9.79) - open/close 465 (time 1786632702.17 total 41.39 last 6.90) - open/close 526 (time 1786632713.22 total 52.44 last 5.52) - open/close 631 (time 1786632724.02 total 63.24 last 9.72) - open/close 747 (time 1786632734.88 total 74.10 last 10.68) - open/close 914 (time 1786632744.89 total 84.11 last 16.68) - open/close 1145 (time 1786632754.89 total 94.11 last 23.10) - open/close 1202 (time 1786632765.58 total 104.80 last 5.33) - open/close 1372 (time 1786632775.60 total 114.82 last 16.97) - open/close 1517 (time 1786632785.62 total 124.83 last 14.48) - open/close 1667 (time 1786632795.65 total 134.87 last 14.94) - open/close 1769 (time 1786632806.55 total 145.76 last 9.37) - open/close 1879 (time 1786632817.54 total 156.76 last 10.01) - open/close 2018 (time 1786632827.94 total 167.16 last 13.36) - open/close 2143 (time 1786632838.08 total 177.30 last 12.32) - open/close 2249 (time 1786632848.72 total 187.94 last 9.97) - open/close 2325 (time 1786632860.93 total 200.15 last 6.22) - open/close 2470 (time 1786632871.51 total 210.73 last 13.71) - open/close 2749 (time 1786632882.59 total 221.81 last 25.18) - open/close 2897 (time 1786632893.36 total 232.58 last 13.74) - open/close 3121 (time 1786632904.02 total 243.24 last 21.02) - open/close 3344 (time 1786632914.35 total 253.56 last 21.59) - open/close 3559 (time 1786632924.95 total 264.17 last 20.27) - open/close 3631 (time 1786632935.19 total 274.41 last 7.03) - open/close 3694 (time 1786632946.38 total 285.59 last 5.63) - open/close 3836 (time 1786632956.38 total 295.60 last 14.20) - open/close 3900 (time 1786632967.08 total 306.30 last 5.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-3953) error: No space left on device total: 3953 open/close in 307.50 seconds: 12.86 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (114881) - No such process - unlinked 0 (time 1786632976 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-3953) error: No such file or directory total: 3953 unlinks in 30 seconds: 131.766663 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg117-server unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: modules unloaded. pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2