== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:16:12 (1781684172) start mds service on oleg431-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 612 (time 1781684215.37 total 10.98 last 55.72) - open/close 1314 (time 1781684226.38 total 21.99 last 63.76) - open/close 1701 (time 1781684236.56 total 32.17 last 38.03) - open/close 2268 (time 1781684247.12 total 42.73 last 53.70) - open/close 2781 (time 1781684257.62 total 53.23 last 48.85) - open/close 3408 (time 1781684268.56 total 64.17 last 57.32) - open/close 4009 (time 1781684279.31 total 74.92 last 55.89) - open/close 4568 (time 1781684289.87 total 85.48 last 52.94) - open/close 5103 (time 1781684300.43 total 96.04 last 50.67) - open/close 5673 (time 1781684311.18 total 106.79 last 53.01) - open/close 6349 (time 1781684321.18 total 116.80 last 67.58) - open/close 6946 (time 1781684331.21 total 126.82 last 59.54) - open/close 7585 (time 1781684341.32 total 136.94 last 63.19) - open/close 8203 (time 1781684352.33 total 147.95 last 56.14) - open/close 8742 (time 1781684363.02 total 158.64 last 50.42) - open/close 9332 (time 1781684373.02 total 168.64 last 59.00) - open/close 9915 (time 1781684383.24 total 178.86 last 57.04) - open/close 10507 (time 1781684393.24 total 188.86 last 59.20) - open/close 11120 (time 1781684403.25 total 198.86 last 61.28) - open/close 11743 (time 1781684414.09 total 209.71 last 57.44) - open/close 12376 (time 1781684424.91 total 220.52 last 58.54) - open/close 13035 (time 1781684435.15 total 230.76 last 64.34) - open/close 13632 (time 1781684445.39 total 241.00 last 58.30) - open/close 14243 (time 1781684455.39 total 251.01 last 61.08) - open/close 14850 (time 1781684465.42 total 261.03 last 60.54) - open/close 15486 (time 1781684475.42 total 271.04 last 63.59) - open/close 16021 (time 1781684485.96 total 281.58 last 50.75) - open/close 16470 (time 1781684496.59 total 292.20 last 42.25) - open/close 16978 (time 1781684507.02 total 302.63 last 48.71) - open/close 20000 (time 1781684514.56 total 310.17 last 400.91) - open/close 24366 (time 1781684524.56 total 320.17 last 436.54) - open/close 28766 (time 1781684534.56 total 330.17 last 439.95) - open/close 30000 (time 1781684537.39 total 333.01 last 435.25) - open/close 34064 (time 1781684547.40 total 343.01 last 406.35) - open/close 39000 (time 1781684557.40 total 353.01 last 493.54) - open/close 40000 (time 1781684559.60 total 355.22 last 453.13) - open/close 44753 (time 1781684569.60 total 365.22 last 475.30) - open/close 49542 (time 1781684579.61 total 375.22 last 478.85) total: 50000 open/close in 376.17 seconds: 132.92 ops/second - unlinked 0 (time 1781684581 ; total 0 ; last 0) - unlinked 10000 (time 1781684596 ; total 15 ; last 15) - unlinked 20000 (time 1781684613 ; total 32 ; last 17) - unlinked 30000 (time 1781684628 ; total 47 ; last 15) - unlinked 40000 (time 1781684643 ; total 62 ; last 15) total: 50000 unlinks in 78 seconds: 641.025635 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: modules unloaded. pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2