== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:56:29 (1778349389) start mds service on oleg412-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg412-server: detected 4 online CPUs by sysfs oleg412-server: MODOPTS_LIBCFS= oleg412-server: Force libcfs to create 2 CPU partitions oleg412-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg412-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg412-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg412-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg412-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 676 (time 1778349447.76 total 10.77 last 62.78) - open/close 1346 (time 1778349458.71 total 21.72 last 61.19) - open/close 1733 (time 1778349469.14 total 32.14 last 37.11) - open/close 2110 (time 1778349479.38 total 42.38 last 36.82) - open/close 2701 (time 1778349490.20 total 53.20 last 54.63) - open/close 3176 (time 1778349500.69 total 63.70 last 45.26) - open/close 3661 (time 1778349511.51 total 74.52 last 44.83) - open/close 4265 (time 1778349521.51 total 84.52 last 60.40) - open/close 4792 (time 1778349532.44 total 95.44 last 48.24) - open/close 5502 (time 1778349543.12 total 106.13 last 66.44) - open/close 6054 (time 1778349553.88 total 116.88 last 51.34) - open/close 6638 (time 1778349564.25 total 127.25 last 56.30) - open/close 7336 (time 1778349574.36 total 137.36 last 69.05) - open/close 7665 (time 1778349584.60 total 147.60 last 32.13) - open/close 8209 (time 1778349594.96 total 157.97 last 52.48) - open/close 8925 (time 1778349605.97 total 168.98 last 65.05) - open/close 9561 (time 1778349616.21 total 179.22 last 62.10) - open/close 10000 (time 1778349624.99 total 188.00 last 49.99) - open/close 10628 (time 1778349635.09 total 198.10 last 62.19) - open/close 11064 (time 1778349645.52 total 208.53 last 41.79) - open/close 11522 (time 1778349656.34 total 219.35 last 42.32) - open/close 11993 (time 1778349666.39 total 229.39 last 46.90) - open/close 12495 (time 1778349676.44 total 239.44 last 49.95) - open/close 12989 (time 1778349686.44 total 249.44 last 49.39) - open/close 13457 (time 1778349697.37 total 260.37 last 42.83) - open/close 13928 (time 1778349707.47 total 270.48 last 46.60) - open/close 14521 (time 1778349717.48 total 280.48 last 59.29) - open/close 14910 (time 1778349728.21 total 291.22 last 36.23) - open/close 15492 (time 1778349739.10 total 302.10 last 53.47) - open/close 16007 (time 1778349749.85 total 312.85 last 47.90) - open/close 16453 (time 1778349760.60 total 323.61 last 41.48) - open/close 16966 (time 1778349770.64 total 333.65 last 51.08) - open/close 17768 (time 1778349780.95 total 343.96 last 77.81) - open/close 18433 (time 1778349790.95 total 353.96 last 66.49) - open/close 19056 (time 1778349801.88 total 364.89 last 57.00) - open/close 20000 (time 1778349805.44 total 368.45 last 265.31) - open/close 23137 (time 1778349815.44 total 378.45 last 313.65) - open/close 26451 (time 1778349825.44 total 388.45 last 331.34) - open/close 29522 (time 1778349835.45 total 398.45 last 307.07) - open/close 32978 (time 1778349845.45 total 408.45 last 345.47) - open/close 36070 (time 1778349855.45 total 418.46 last 309.16) - open/close 39642 (time 1778349865.45 total 428.46 last 357.18) - open/close 42884 (time 1778349875.45 total 438.46 last 324.11) - open/close 46324 (time 1778349885.46 total 448.46 last 343.95) - open/close 49436 (time 1778349895.46 total 458.46 last 311.16) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 459.13 seconds: 108.10 ops/second - unlinked 0 (time 1778349897 ; total 0 ; last 0) - unlinked 10000 (time 1778349912 ; total 15 ; last 15) - unlinked 20000 (time 1778349930 ; total 33 ; last 18) - unlinked 30000 (time 1778349948 ; total 51 ; last 18) - unlinked 40000 (time 1778349965 ; total 68 ; last 17) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 84 seconds: 590.857117 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules via unload_modules_local on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local oleg412-server: modules unloaded. pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2