== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:18:06 (1780809486) start mds service on oleg324-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg324-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg324-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg324-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 408 (time 1780809525.47 total 10.17 last 40.13) - open/close 884 (time 1780809535.78 total 20.47 last 46.19) - open/close 1478 (time 1780809546.47 total 31.16 last 55.58) - open/close 2086 (time 1780809557.22 total 41.91 last 56.56) - open/close 2502 (time 1780809567.46 total 52.15 last 40.62) - open/close 3052 (time 1780809577.89 total 62.58 last 52.73) - open/close 3402 (time 1780809587.94 total 72.63 last 34.83) - open/close 4162 (time 1780809598.95 total 83.64 last 69.04) - open/close 4696 (time 1780809609.57 total 94.26 last 50.27) - open/close 5045 (time 1780809619.62 total 104.31 last 34.73) - open/close 5428 (time 1780809629.73 total 114.42 last 37.88) - open/close 5825 (time 1780809639.90 total 124.60 last 39.02) - open/close 6209 (time 1780809649.96 total 134.65 last 38.20) - open/close 6899 (time 1780809660.90 total 145.59 last 63.06) - open/close 7723 (time 1780809671.14 total 155.83 last 80.48) - open/close 8136 (time 1780809681.31 total 166.01 last 40.59) - open/close 8753 (time 1780809692.13 total 176.82 last 57.03) - open/close 9345 (time 1780809702.94 total 187.64 last 54.75) - open/close 9655 (time 1780809712.95 total 197.64 last 31.00) - open/close 10000 (time 1780809718.85 total 203.54 last 58.46) - open/close 10682 (time 1780809729.89 total 214.58 last 61.76) - open/close 11289 (time 1780809740.77 total 225.46 last 55.79) - open/close 11820 (time 1780809751.27 total 235.96 last 50.58) - open/close 12557 (time 1780809762.27 total 246.97 last 66.97) - open/close 13130 (time 1780809772.96 total 257.66 last 53.60) - open/close 13650 (time 1780809783.46 total 268.15 last 49.56) - open/close 14280 (time 1780809793.51 total 278.20 last 62.69) - open/close 14894 (time 1780809804.39 total 289.08 last 56.44) - open/close 17982 (time 1780809814.39 total 299.08 last 308.79) - open/close 20000 (time 1780809818.48 total 303.17 last 492.96) - open/close 24819 (time 1780809828.48 total 313.17 last 481.87) - open/close 29442 (time 1780809838.48 total 323.17 last 462.22) - open/close 34346 (time 1780809848.48 total 333.18 last 490.35) - open/close 39568 (time 1780809858.48 total 343.18 last 522.11) - open/close 44308 (time 1780809868.49 total 353.18 last 473.83) - open/close 49108 (time 1780809878.49 total 363.18 last 479.95) total: 50000 open/close in 364.85 seconds: 137.04 ops/second - unlinked 0 (time 1780809881 ; total 0 ; last 0) - unlinked 10000 (time 1780809896 ; total 15 ; last 15) - unlinked 20000 (time 1780809909 ; total 28 ; last 13) - unlinked 30000 (time 1780809922 ; total 41 ; last 13) - unlinked 40000 (time 1780809935 ; total 54 ; last 13) total: 50000 unlinks in 68 seconds: 735.294128 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2