== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:42:58 (1773103378) start mds service on oleg409-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions oleg409-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg409-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg409-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 719 (time 1773103419.07 total 10.01 last 71.86) - open/close 1342 (time 1773103429.12 total 20.06 last 61.99) - open/close 1778 (time 1773103439.55 total 30.49 last 41.80) - open/close 2488 (time 1773103449.56 total 40.49 last 70.98) - open/close 3081 (time 1773103460.55 total 51.48 last 53.95) - open/close 3697 (time 1773103470.72 total 61.65 last 60.55) - open/close 4272 (time 1773103480.77 total 71.71 last 57.20) - open/close 4820 (time 1773103490.88 total 81.82 last 54.21) - open/close 5336 (time 1773103501.00 total 91.93 last 51.02) - open/close 5920 (time 1773103511.00 total 101.93 last 58.39) - open/close 6526 (time 1773103521.79 total 112.73 last 56.13) - open/close 7018 (time 1773103532.22 total 123.16 last 47.17) - open/close 7569 (time 1773103542.92 total 133.85 last 51.54) - open/close 8151 (time 1773103553.67 total 144.60 last 54.14) - open/close 8768 (time 1773103563.67 total 154.60 last 61.68) - open/close 9194 (time 1773103574.08 total 165.01 last 40.91) - open/close 9695 (time 1773103584.53 total 175.46 last 47.97) - open/close 10000 (time 1773103588.52 total 179.45 last 76.33) - open/close 10593 (time 1773103599.24 total 190.17 last 55.35) - open/close 11178 (time 1773103609.98 total 200.92 last 54.42) - open/close 11623 (time 1773103620.36 total 211.29 last 42.90) - open/close 12309 (time 1773103631.30 total 222.23 last 62.69) - open/close 12862 (time 1773103641.92 total 232.86 last 52.05) - open/close 13383 (time 1773103652.61 total 243.55 last 48.74) - open/close 13869 (time 1773103662.61 total 253.55 last 48.59) - open/close 14469 (time 1773103672.61 total 263.55 last 60.00) - open/close 14985 (time 1773103683.40 total 274.33 last 47.85) - open/close 15397 (time 1773103693.41 total 284.34 last 41.17) - open/close 15761 (time 1773103703.49 total 294.43 last 36.08) - open/close 16107 (time 1773103714.38 total 305.31 last 31.78) - open/close 18102 (time 1773103724.38 total 315.31 last 199.49) - open/close 20000 (time 1773103730.05 total 320.98 last 334.93) - open/close 23789 (time 1773103740.05 total 330.98 last 378.89) - open/close 26234 (time 1773103750.05 total 340.98 last 244.45) - open/close 28542 (time 1773103760.05 total 350.99 last 230.74) - open/close 30000 (time 1773103764.79 total 355.72 last 308.05) - open/close 34060 (time 1773103774.79 total 365.72 last 405.95) - open/close 37616 (time 1773103784.80 total 375.73 last 355.35) - open/close 40000 (time 1773103791.83 total 382.76 last 338.86) - open/close 43273 (time 1773103801.83 total 392.77 last 327.21) - open/close 47133 (time 1773103811.83 total 402.77 last 385.98) total: 50000 open/close in 409.21 seconds: 122.19 ops/second - unlinked 0 (time 1773103819 ; total 0 ; last 0) - unlinked 10000 (time 1773103835 ; total 16 ; last 16) - unlinked 20000 (time 1773103851 ; total 32 ; last 16) - unlinked 30000 (time 1773103869 ; total 50 ; last 18) - unlinked 40000 (time 1773103884 ; total 65 ; last 15) total: 50000 unlinks in 77 seconds: 649.350647 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2