== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:36:05 (1778279765) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 726 (time 1778279804.79 total 10.00 last 72.60) - open/close 1439 (time 1778279815.61 total 20.82 last 65.88) - open/close 2163 (time 1778279825.72 total 30.94 last 71.60) - open/close 2926 (time 1778279836.66 total 41.88 last 69.73) - open/close 3584 (time 1778279847.61 total 52.82 last 60.12) - open/close 3988 (time 1778279857.72 total 62.94 last 39.95) - open/close 4483 (time 1778279868.15 total 73.37 last 47.46) - open/close 4827 (time 1778279879.16 total 84.37 last 31.25) - open/close 5411 (time 1778279889.91 total 95.13 last 54.31) - open/close 6062 (time 1778279900.86 total 106.07 last 59.49) - open/close 6733 (time 1778279910.86 total 116.07 last 67.09) - open/close 7466 (time 1778279920.86 total 126.07 last 73.30) - open/close 8090 (time 1778279931.58 total 136.79 last 58.22) - open/close 8698 (time 1778279942.33 total 147.54 last 56.55) - open/close 9445 (time 1778279953.34 total 158.55 last 67.85) - open/close 10000 (time 1778279961.87 total 167.08 last 65.09) - open/close 10618 (time 1778279972.73 total 177.94 last 56.89) - open/close 11318 (time 1778279983.74 total 188.95 last 63.58) - open/close 12028 (time 1778279994.74 total 199.96 last 64.51) - open/close 12651 (time 1778280005.56 total 210.78 last 57.59) - open/close 13247 (time 1778280016.31 total 221.53 last 55.44) - open/close 13905 (time 1778280026.31 total 231.53 last 65.79) - open/close 14472 (time 1778280036.92 total 242.13 last 53.45) - open/close 15058 (time 1778280047.61 total 252.83 last 54.81) - open/close 15671 (time 1778280058.30 total 263.51 last 57.38) - open/close 16362 (time 1778280069.18 total 274.39 last 63.51) - open/close 17059 (time 1778280079.18 total 284.39 last 69.70) - open/close 17661 (time 1778280089.92 total 295.13 last 56.06) - open/close 19776 (time 1778280099.92 total 305.13 last 211.48) - open/close 24396 (time 1778280109.92 total 315.13 last 461.99) - open/close 28938 (time 1778280119.92 total 325.13 last 454.12) - open/close 30000 (time 1778280122.24 total 327.45 last 457.99) - open/close 34330 (time 1778280132.24 total 337.45 last 432.92) - open/close 38789 (time 1778280142.24 total 347.45 last 445.84) - open/close 40000 (time 1778280144.94 total 350.15 last 449.37) - open/close 44790 (time 1778280154.94 total 360.15 last 478.91) - open/close 49507 (time 1778280164.94 total 370.15 last 471.65) total: 50000 open/close in 371.07 seconds: 134.74 ops/second - unlinked 0 (time 1778280167 ; total 0 ; last 0) - unlinked 10000 (time 1778280181 ; total 14 ; last 14) - unlinked 20000 (time 1778280195 ; total 28 ; last 14) - unlinked 30000 (time 1778280208 ; total 41 ; last 13) - unlinked 40000 (time 1778280222 ; total 55 ; last 14) total: 50000 unlinks in 69 seconds: 724.637695 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2