== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:05:21 (1773669921) start mds service on oleg427-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 266 (time 1773669963.85 total 10.53 last 25.25) - open/close 587 (time 1773669974.41 total 21.10 last 30.39) - open/close 1048 (time 1773669985.35 total 32.04 last 42.14) - open/close 1340 (time 1773669997.70 total 44.39 last 23.64) - open/close 1799 (time 1773670007.76 total 54.44 last 45.64) - open/close 2165 (time 1773670018.57 total 65.25 last 33.87) - open/close 2452 (time 1773670028.57 total 75.25 last 28.69) - open/close 2602 (time 1773670038.92 total 85.60 last 14.49) - open/close 2708 (time 1773670050.83 total 97.51 last 8.90) - open/close 2892 (time 1773670063.89 total 110.57 last 14.09) - open/close 3069 (time 1773670074.06 total 120.74 last 17.41) - open/close 3294 (time 1773670084.23 total 130.91 last 22.11) - open/close 3759 (time 1773670094.28 total 140.96 last 46.27) - open/close 4319 (time 1773670104.58 total 151.27 last 54.36) - open/close 4578 (time 1773670114.95 total 161.63 last 24.98) - open/close 4865 (time 1773670125.26 total 171.94 last 27.84) - open/close 5077 (time 1773670135.30 total 181.99 last 21.11) - open/close 5233 (time 1773670145.42 total 192.10 last 15.42) - open/close 5765 (time 1773670155.42 total 202.11 last 53.19) - open/close 5940 (time 1773670166.47 total 213.15 last 15.84) - open/close 6397 (time 1773670178.70 total 225.38 last 37.37) - open/close 6668 (time 1773670189.26 total 235.94 last 25.67) - open/close 6826 (time 1773670199.50 total 246.18 last 15.43) - open/close 6966 (time 1773670209.61 total 256.29 last 13.84) - open/close 7091 (time 1773670219.66 total 266.34 last 12.44) - open/close 7570 (time 1773670229.66 total 276.34 last 47.90) - open/close 8229 (time 1773670240.21 total 286.89 last 62.48) - open/close 8985 (time 1773670250.76 total 297.44 last 71.62) - open/close 9554 (time 1773670261.06 total 307.75 last 55.24) - open/close 10000 (time 1773670269.69 total 316.37 last 51.71) - open/close 10481 (time 1773670279.82 total 326.50 last 47.47) - open/close 10808 (time 1773670289.87 total 336.55 last 32.55) - open/close 10976 (time 1773670299.98 total 346.66 last 16.61) - open/close 12950 (time 1773670309.98 total 356.66 last 197.36) - open/close 16714 (time 1773670319.98 total 366.66 last 376.38) - open/close 20000 (time 1773670328.88 total 375.56 last 369.28) - open/close 23359 (time 1773670338.88 total 385.57 last 335.79) - open/close 27516 (time 1773670348.89 total 395.57 last 415.61) - open/close 30000 (time 1773670354.91 total 401.59 last 412.42) - open/close 33272 (time 1773670364.91 total 411.59 last 327.18) - open/close 36886 (time 1773670374.91 total 421.59 last 361.37) - open/close 40000 (time 1773670383.32 total 430.00 last 370.23) - open/close 43179 (time 1773670393.32 total 440.01 last 317.83) - open/close 46638 (time 1773670403.33 total 450.01 last 345.86) total: 50000 open/close in 458.82 seconds: 108.98 ops/second - unlinked 0 (time 1773670413 ; total 0 ; last 0) - unlinked 10000 (time 1773670428 ; total 15 ; last 15) - unlinked 20000 (time 1773670442 ; total 29 ; last 14) - unlinked 30000 (time 1773670457 ; total 44 ; last 15) - unlinked 40000 (time 1773670470 ; total 57 ; last 13) total: 50000 unlinks in 72 seconds: 694.444458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2