== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:14:08 (1771402448) start mds service on oleg310-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1771402503.83 total 10.58 last 9.93) - open/close 269 (time 1771402514.07 total 20.82 last 16.01) - open/close 449 (time 1771402524.75 total 31.50 last 16.85) - open/close 575 (time 1771402535.57 total 42.32 last 11.64) - open/close 705 (time 1771402545.58 total 52.33 last 12.99) - open/close 836 (time 1771402556.63 total 63.38 last 11.86) - open/close 1084 (time 1771402566.63 total 73.38 last 24.80) - open/close 1279 (time 1771402576.66 total 83.41 last 19.43) - open/close 1453 (time 1771402586.67 total 93.42 last 17.39) - open/close 1621 (time 1771402596.67 total 103.42 last 16.80) - open/close 1708 (time 1771402607.19 total 113.94 last 8.27) - open/close 1999 (time 1771402617.19 total 123.94 last 29.10) - open/close 2260 (time 1771402627.19 total 133.94 last 26.10) - open/close 2497 (time 1771402638.04 total 144.79 last 21.86) - open/close 2644 (time 1771402648.99 total 155.74 last 13.42) - open/close 2771 (time 1771402659.79 total 166.54 last 11.76) - open/close 2918 (time 1771402669.90 total 176.65 last 14.53) - open/close 3004 (time 1771402680.28 total 187.03 last 8.29) - open/close 3131 (time 1771402690.98 total 197.72 last 11.88) - open/close 3307 (time 1771402701.40 total 208.15 last 16.88) - open/close 3410 (time 1771402712.99 total 219.74 last 8.89) - open/close 3668 (time 1771402723.92 total 230.67 last 23.59) - open/close 3936 (time 1771402734.74 total 241.49 last 24.76) - open/close 4157 (time 1771402745.04 total 251.79 last 21.47) - open/close 4329 (time 1771402755.04 total 261.79 last 17.20) - open/close 4498 (time 1771402765.46 total 272.21 last 16.22) - open/close 4650 (time 1771402775.47 total 282.22 last 15.19) - open/close 4739 (time 1771402785.94 total 292.69 last 8.50) - open/close 4941 (time 1771402796.50 total 303.25 last 19.13) - open/close 5078 (time 1771402807.38 total 314.13 last 12.59) - open/close 5232 (time 1771402818.19 total 324.94 last 14.25) - open/close 5553 (time 1771402828.19 total 334.94 last 32.09) - open/close 5826 (time 1771402838.79 total 345.54 last 25.75) - open/close 6022 (time 1771402848.80 total 355.55 last 19.58) - open/close 6263 (time 1771402859.09 total 365.83 last 23.44) - open/close 6440 (time 1771402869.09 total 375.84 last 17.69) - open/close 6690 (time 1771402880.08 total 386.83 last 22.75) - open/close 6932 (time 1771402891.48 total 398.23 last 21.23) - open/close 7221 (time 1771402901.48 total 408.23 last 28.89) - open/close 7465 (time 1771402911.75 total 418.50 last 23.75) - open/close 7735 (time 1771402922.14 total 428.88 last 26.01) - open/close 8008 (time 1771402932.45 total 439.20 last 26.47) - open/close 8201 (time 1771402942.87 total 449.62 last 18.52) - open/close 8476 (time 1771402953.81 total 460.56 last 25.13) - open/close 8720 (time 1771402964.50 total 471.25 last 22.83) - open/close 8966 (time 1771402974.68 total 481.43 last 24.18) - open/close 9254 (time 1771402985.23 total 491.98 last 27.28) - open/close 9521 (time 1771402995.79 total 502.54 last 25.28) - open/close 9846 (time 1771403006.80 total 513.55 last 29.54) - open/close 10000 (time 1771403010.07 total 516.81 last 47.13) - open/close 10542 (time 1771403020.07 total 526.82 last 54.17) - open/close 12383 (time 1771403030.08 total 536.82 last 184.02) - open/close 14185 (time 1771403040.08 total 546.83 last 180.13) - open/close 16028 (time 1771403050.08 total 556.83 last 184.26) - open/close 17937 (time 1771403060.08 total 566.83 last 190.86) - open/close 19800 (time 1771403070.09 total 576.84 last 186.21) - open/close 21485 (time 1771403080.09 total 586.84 last 168.42) - open/close 23559 (time 1771403090.10 total 596.85 last 207.34) - open/close 25509 (time 1771403100.10 total 606.85 last 194.96) - open/close 27635 (time 1771403110.10 total 616.85 last 212.60) - open/close 29789 (time 1771403120.10 total 626.85 last 215.35) - open/close 31506 (time 1771403130.11 total 636.85 last 171.62) - open/close 33003 (time 1771403140.11 total 646.86 last 149.65) - open/close 34782 (time 1771403150.11 total 656.86 last 177.83) - open/close 36679 (time 1771403160.12 total 666.87 last 189.57) - open/close 38309 (time 1771403170.12 total 676.87 last 162.96) - open/close 40000 (time 1771403178.74 total 685.49 last 196.15) - open/close 42073 (time 1771403188.75 total 695.50 last 207.19) - open/close 43751 (time 1771403198.75 total 705.50 last 167.77) - open/close 45331 (time 1771403208.75 total 715.50 last 157.94) - open/close 47445 (time 1771403218.76 total 725.51 last 211.27) - open/close 49228 (time 1771403228.78 total 735.52 last 178.03) total: 50000 open/close in 740.95 seconds: 67.48 ops/second - unlinked 0 (time 1771403236 ; total 0 ; last 0) - unlinked 10000 (time 1771403274 ; total 38 ; last 38) - unlinked 20000 (time 1771403307 ; total 71 ; last 33) - unlinked 30000 (time 1771403334 ; total 98 ; last 27) - unlinked 40000 (time 1771403370 ; total 134 ; last 36) total: 50000 unlinks in 170 seconds: 294.117645 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: modules unloaded. pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2