== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:49:44 (1780087784) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 339 (time 1780087827.93 total 10.00 last 33.89) - open/close 768 (time 1780087838.12 total 20.19 last 42.10) - open/close 1019 (time 1780087848.24 total 30.31 last 24.80) - open/close 1281 (time 1780087858.54 total 40.61 last 25.44) - open/close 1594 (time 1780087869.55 total 51.62 last 28.43) - open/close 1866 (time 1780087879.85 total 61.93 last 26.39) - open/close 2173 (time 1780087890.47 total 72.55 last 28.91) - open/close 2486 (time 1780087901.23 total 83.30 last 29.11) - open/close 2817 (time 1780087911.85 total 93.92 last 31.16) - open/close 3061 (time 1780087922.80 total 104.87 last 22.29) - open/close 3294 (time 1780087932.80 total 114.87 last 23.29) - open/close 3593 (time 1780087943.66 total 125.73 last 27.53) - open/close 3850 (time 1780087954.16 total 136.24 last 24.47) - open/close 4071 (time 1780087964.65 total 146.73 last 21.07) - open/close 4369 (time 1780087975.21 total 157.28 last 28.23) - open/close 4605 (time 1780087985.26 total 167.33 last 23.49) - open/close 4835 (time 1780087996.01 total 178.08 last 21.40) - open/close 5086 (time 1780088006.57 total 188.64 last 23.76) - open/close 5444 (time 1780088016.81 total 198.88 last 34.97) - open/close 5694 (time 1780088027.05 total 209.12 last 24.41) - open/close 6006 (time 1780088037.88 total 219.95 last 28.82) - open/close 6297 (time 1780088047.97 total 230.05 last 28.82) - open/close 6554 (time 1780088057.97 total 240.05 last 25.70) - open/close 6892 (time 1780088068.01 total 250.08 last 33.70) - open/close 7117 (time 1780088078.83 total 260.90 last 20.79) - open/close 7390 (time 1780088089.07 total 271.14 last 26.66) - open/close 7666 (time 1780088099.94 total 282.02 last 25.38) - open/close 7940 (time 1780088110.20 total 292.27 last 26.71) - open/close 8135 (time 1780088120.55 total 302.63 last 18.84) - open/close 8319 (time 1780088130.80 total 312.87 last 17.96) - open/close 8545 (time 1780088143.66 total 325.73 last 17.57) - open/close 8753 (time 1780088154.09 total 336.16 last 19.95) - open/close 9219 (time 1780088164.09 total 346.16 last 46.60) - open/close 9693 (time 1780088175.02 total 357.09 last 43.37) - open/close 10000 (time 1780088182.44 total 364.51 last 41.37) - open/close 10259 (time 1780088193.19 total 375.26 last 24.09) - open/close 10397 (time 1780088203.24 total 385.31 last 13.73) - open/close 10623 (time 1780088213.81 total 395.88 last 21.39) - open/close 11449 (time 1780088223.81 total 405.88 last 82.58) - open/close 13224 (time 1780088233.81 total 415.89 last 177.40) - open/close 15005 (time 1780088243.82 total 425.89 last 177.99) - open/close 16865 (time 1780088253.82 total 435.90 last 185.97) - open/close 18791 (time 1780088263.82 total 445.90 last 192.56) - open/close 20000 (time 1780088268.80 total 450.88 last 242.79) - open/close 21815 (time 1780088278.81 total 460.88 last 181.46) - open/close 23147 (time 1780088288.81 total 470.88 last 133.15) - open/close 24728 (time 1780088298.81 total 480.89 last 158.05) - open/close 26364 (time 1780088308.81 total 490.89 last 163.57) - open/close 28320 (time 1780088318.82 total 500.89 last 195.55) - open/close 30000 (time 1780088325.16 total 507.23 last 264.82) - open/close 32830 (time 1780088335.16 total 517.24 last 282.94) - open/close 34867 (time 1780088345.16 total 527.24 last 203.67) - open/close 36797 (time 1780088355.17 total 537.24 last 192.97) - open/close 39313 (time 1780088365.17 total 547.24 last 251.60) - open/close 40000 (time 1780088367.94 total 550.01 last 247.84) - open/close 42130 (time 1780088377.94 total 560.01 last 212.98) - open/close 44872 (time 1780088387.94 total 570.01 last 274.18) - open/close 47307 (time 1780088397.94 total 580.01 last 243.46) - open/close 49113 (time 1780088407.94 total 590.02 last 180.57) total: 50000 open/close in 596.20 seconds: 83.86 ops/second - unlinked 0 (time 1780088415 ; total 0 ; last 0) - unlinked 10000 (time 1780088451 ; total 36 ; last 36) - unlinked 20000 (time 1780088485 ; total 70 ; last 34) - unlinked 30000 (time 1780088509 ; total 94 ; last 24) - unlinked 40000 (time 1780088533 ; total 118 ; last 24) total: 50000 unlinks in 143 seconds: 349.650360 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2