== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:35:18 (1781030118) start mds service on oleg430-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg430-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 190 (time 1781030162.56 total 10.58 last 17.96) - open/close 375 (time 1781030173.05 total 21.07 last 17.63) - open/close 630 (time 1781030183.05 total 31.08 last 25.50) - open/close 857 (time 1781030193.06 total 41.08 last 22.69) - open/close 983 (time 1781030204.09 total 52.11 last 11.43) - open/close 1153 (time 1781030214.40 total 62.42 last 16.48) - open/close 1289 (time 1781030224.45 total 72.47 last 13.54) - open/close 1462 (time 1781030234.55 total 82.57 last 17.12) - open/close 1586 (time 1781030244.55 total 92.58 last 12.40) - open/close 1774 (time 1781030254.84 total 102.87 last 18.27) - open/close 2111 (time 1781030264.84 total 112.87 last 33.70) - open/close 2353 (time 1781030275.64 total 123.66 last 22.41) - open/close 2637 (time 1781030286.33 total 134.36 last 26.56) - open/close 2881 (time 1781030296.33 total 144.36 last 24.40) - open/close 3101 (time 1781030307.01 total 155.03 last 20.62) - open/close 3262 (time 1781030317.24 total 165.27 last 15.73) - open/close 3432 (time 1781030327.61 total 175.63 last 16.40) - open/close 3545 (time 1781030338.62 total 186.64 last 10.26) - open/close 3834 (time 1781030349.24 total 197.26 last 27.22) - open/close 4203 (time 1781030359.42 total 207.45 last 36.24) - open/close 4407 (time 1781030370.04 total 218.07 last 19.21) - open/close 4635 (time 1781030380.86 total 228.88 last 21.08) - open/close 4886 (time 1781030391.35 total 239.37 last 23.92) - open/close 5069 (time 1781030401.53 total 249.56 last 17.97) - open/close 5205 (time 1781030411.64 total 259.67 last 13.45) - open/close 5435 (time 1781030421.88 total 269.90 last 22.47) - open/close 5718 (time 1781030432.51 total 280.53 last 26.64) - open/close 5964 (time 1781030442.87 total 290.89 last 23.73) - open/close 6095 (time 1781030452.92 total 300.94 last 13.04) - open/close 6272 (time 1781030463.42 total 311.44 last 16.86) - open/close 6430 (time 1781030473.78 total 321.81 last 15.25) - open/close 6684 (time 1781030484.60 total 332.62 last 23.49) - open/close 6796 (time 1781030494.71 total 342.74 last 11.07) - open/close 7061 (time 1781030505.73 total 353.75 last 24.06) - open/close 7222 (time 1781030516.22 total 364.24 last 15.34) - open/close 7492 (time 1781030526.23 total 374.25 last 26.99) - open/close 8108 (time 1781030536.23 total 384.25 last 61.58) - open/close 9852 (time 1781030546.23 total 394.25 last 174.39) - open/close 11452 (time 1781030556.23 total 404.25 last 159.99) - open/close 13224 (time 1781030566.23 total 414.26 last 177.14) - open/close 15250 (time 1781030576.24 total 424.26 last 202.55) - open/close 16945 (time 1781030586.24 total 434.26 last 169.44) - open/close 18594 (time 1781030596.24 total 444.26 last 164.86) - open/close 20000 (time 1781030605.09 total 453.11 last 158.88) - open/close 21893 (time 1781030615.09 total 463.11 last 189.30) - open/close 23712 (time 1781030625.09 total 473.12 last 181.84) - open/close 25795 (time 1781030635.10 total 483.12 last 208.24) - open/close 27823 (time 1781030645.10 total 493.12 last 202.79) - open/close 29221 (time 1781030655.10 total 503.12 last 139.74) - open/close 30000 (time 1781030660.43 total 508.45 last 146.22) - open/close 31447 (time 1781030670.43 total 518.45 last 144.68) - open/close 32933 (time 1781030680.44 total 528.46 last 148.55) - open/close 34256 (time 1781030690.44 total 538.46 last 132.22) - open/close 35776 (time 1781030700.44 total 548.47 last 151.98) - open/close 37389 (time 1781030710.44 total 558.47 last 161.28) - open/close 38977 (time 1781030720.45 total 568.47 last 158.73) - open/close 40000 (time 1781030725.44 total 573.46 last 204.99) - open/close 42456 (time 1781030735.44 total 583.46 last 245.58) - open/close 45249 (time 1781030745.44 total 593.46 last 279.26) - open/close 48425 (time 1781030755.44 total 603.47 last 317.53) total: 50000 open/close in 608.02 seconds: 82.23 ops/second - unlinked 0 (time 1781030761 ; total 0 ; last 0) - unlinked 10000 (time 1781030786 ; total 25 ; last 25) - unlinked 20000 (time 1781030812 ; total 51 ; last 26) - unlinked 30000 (time 1781030837 ; total 76 ; last 25) - unlinked 40000 (time 1781030866 ; total 105 ; last 29) total: 50000 unlinks in 136 seconds: 367.647064 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local oleg430-server: modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2