== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:24:13 (1784406253) start mds service on oleg143-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing load_modules_local oleg143-server: Loading modules from /home/green/git/lustre-release/lustre oleg143-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg143-server: detected 4 online CPUs by sysfs oleg143-server: MODOPTS_LIBCFS= oleg143-server: Force libcfs to create 2 CPU partitions oleg143-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg143-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg143-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg143-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg143-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg143-server: oleg143-server.virtnet: executing set_default_debug -1 all pdsh@oleg143-client: oleg143-server: ssh exited with exit code 1 Started lustre-OST0000 oleg143-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg143-client.virtnet: -o user_xattr,flock 192.168.201.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 369 (time 1784406295.53 total 10.82 last 34.11) - open/close 634 (time 1784406305.97 total 21.26 last 25.39) - open/close 780 (time 1784406316.20 total 31.49 last 14.27) - open/close 974 (time 1784406326.76 total 42.05 last 18.37) - open/close 1404 (time 1784406336.76 total 52.05 last 42.99) - open/close 1729 (time 1784406347.76 total 63.05 last 29.55) - open/close 2016 (time 1784406358.57 total 73.86 last 26.55) - open/close 2260 (time 1784406369.00 total 84.29 last 23.38) - open/close 2643 (time 1784406380.07 total 95.36 last 34.60) - open/close 3064 (time 1784406390.19 total 105.47 last 41.64) - open/close 3488 (time 1784406400.19 total 115.47 last 42.39) - open/close 3946 (time 1784406410.22 total 125.50 last 45.66) - open/close 4463 (time 1784406421.42 total 136.71 last 46.14) - open/close 5095 (time 1784406431.53 total 146.81 last 62.53) - open/close 5468 (time 1784406442.54 total 157.83 last 33.87) - open/close 5778 (time 1784406453.30 total 168.59 last 28.80) - open/close 6020 (time 1784406463.85 total 179.14 last 22.94) - open/close 6465 (time 1784406474.79 total 190.08 last 40.66) - open/close 6840 (time 1784406485.55 total 200.83 last 34.88) - open/close 7464 (time 1784406495.55 total 210.83 last 62.39) - open/close 8029 (time 1784406506.35 total 221.63 last 52.32) - open/close 8580 (time 1784406517.23 total 232.51 last 50.65) - open/close 9021 (time 1784406527.53 total 242.82 last 42.79) - open/close 9475 (time 1784406537.71 total 252.99 last 44.63) - open/close 9910 (time 1784406547.95 total 263.23 last 42.48) - open/close 10536 (time 1784406558.06 total 273.34 last 61.91) - open/close 11123 (time 1784406569.07 total 284.35 last 53.32) - open/close 11728 (time 1784406579.95 total 295.23 last 55.60) - open/close 12324 (time 1784406590.89 total 306.18 last 54.47) - open/close 12832 (time 1784406601.26 total 316.54 last 49.01) - open/close 13245 (time 1784406611.69 total 326.98 last 39.59) - open/close 16607 (time 1784406621.69 total 336.98 last 336.20) - open/close 20000 (time 1784406628.93 total 344.22 last 468.53) - open/close 24849 (time 1784406638.93 total 354.22 last 484.87) - open/close 29247 (time 1784406648.93 total 364.22 last 439.78) - open/close 34003 (time 1784406658.93 total 374.22 last 475.59) - open/close 38708 (time 1784406668.93 total 384.22 last 470.47) - open/close 40000 (time 1784406671.80 total 387.08 last 451.12) - open/close 44457 (time 1784406681.80 total 397.08 last 445.63) - open/close 48990 (time 1784406691.80 total 407.09 last 453.25) total: 50000 open/close in 408.99 seconds: 122.25 ops/second - unlinked 0 (time 1784406694 ; total 0 ; last 0) - unlinked 10000 (time 1784406708 ; total 14 ; last 14) - unlinked 20000 (time 1784406722 ; total 28 ; last 14) - unlinked 30000 (time 1784406737 ; total 43 ; last 15) - unlinked 40000 (time 1784406751 ; total 57 ; last 14) total: 50000 unlinks in 72 seconds: 694.444458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg143-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg143-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg143-server stop mds service on oleg143-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg143-server unloading modules via unload_modules_local on: 'oleg143-server' oleg143-server: oleg143-server.virtnet: executing unload_modules_local oleg143-server: modules unloaded. pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-client: ssh exited with exit code 2 pdsh@oleg143-client: oleg143-server: ssh exited with exit code 2