== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:02:09 (1786172529) start mds service on oleg311-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg311-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg311-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1786172607.45 total 10.20 last 17.06) - open/close 339 (time 1786172617.45 total 20.20 last 16.50) - open/close 589 (time 1786172627.59 total 30.34 last 24.67) - open/close 852 (time 1786172637.82 total 40.57 last 25.71) - open/close 1062 (time 1786172647.85 total 50.60 last 20.92) - open/close 1264 (time 1786172657.94 total 60.68 last 20.04) - open/close 1544 (time 1786172668.69 total 71.44 last 26.02) - open/close 1807 (time 1786172679.22 total 81.96 last 25.00) - open/close 1998 (time 1786172689.69 total 92.44 last 18.23) - open/close 2162 (time 1786172700.27 total 103.02 last 15.50) - open/close 2320 (time 1786172710.47 total 113.22 last 15.49) - open/close 2453 (time 1786172720.51 total 123.25 last 13.25) - open/close 2613 (time 1786172731.04 total 133.79 last 15.19) - open/close 2883 (time 1786172741.53 total 144.28 last 25.75) - open/close 3088 (time 1786172751.54 total 154.28 last 20.48) - open/close 3227 (time 1786172762.52 total 165.27 last 12.65) - open/close 3472 (time 1786172772.58 total 175.33 last 24.36) - open/close 3691 (time 1786172782.97 total 185.71 last 21.09) - open/close 3953 (time 1786172794.03 total 196.78 last 23.68) - open/close 4189 (time 1786172804.69 total 207.44 last 22.13) - open/close 4422 (time 1786172815.25 total 218.00 last 22.08) - open/close 4658 (time 1786172825.58 total 228.33 last 22.84) - open/close 4763 (time 1786172836.19 total 238.94 last 9.90) - open/close 4933 (time 1786172846.26 total 249.01 last 16.88) - open/close 5141 (time 1786172858.14 total 260.89 last 17.51) - open/close 5384 (time 1786172868.78 total 271.53 last 22.85) - open/close 5609 (time 1786172878.80 total 281.54 last 22.46) - open/close 5792 (time 1786172889.77 total 292.51 last 16.68) - open/close 6032 (time 1786172899.78 total 302.53 last 23.97) - open/close 6221 (time 1786172909.81 total 312.56 last 18.84) - open/close 6408 (time 1786172920.62 total 323.37 last 17.30) - open/close 6710 (time 1786172930.64 total 333.39 last 30.15) - open/close 6960 (time 1786172941.54 total 344.29 last 22.94) - open/close 7143 (time 1786172951.63 total 354.38 last 18.13) - open/close 7330 (time 1786172962.11 total 364.86 last 17.85) - open/close 7482 (time 1786172972.14 total 374.89 last 15.15) - open/close 7651 (time 1786172982.70 total 385.45 last 16.01) - open/close 7880 (time 1786172992.71 total 395.46 last 22.88) - open/close 8087 (time 1786173002.73 total 405.48 last 20.67) - open/close 8300 (time 1786173012.76 total 415.51 last 21.23) - open/close 8475 (time 1786173023.67 total 426.42 last 16.04) - open/close 8676 (time 1786173033.96 total 436.71 last 19.54) - open/close 8891 (time 1786173044.96 total 447.71 last 19.54) - open/close 9087 (time 1786173054.98 total 457.72 last 19.57) - open/close 9271 (time 1786173065.97 total 468.72 last 16.73) - open/close 9495 (time 1786173077.02 total 479.77 last 20.28) - open/close 9739 (time 1786173088.04 total 490.78 last 22.14) - open/close 9862 (time 1786173098.04 total 500.79 last 12.29) - open/close 10000 (time 1786173103.84 total 506.59 last 23.79) - open/close 10205 (time 1786173114.19 total 516.94 last 19.81) - open/close 10443 (time 1786173127.08 total 529.83 last 18.46) - open/close 10633 (time 1786173137.09 total 539.83 last 18.99) - open/close 10766 (time 1786173147.18 total 549.93 last 13.18) - open/close 10912 (time 1786173157.19 total 559.94 last 14.58) - open/close 11136 (time 1786173167.20 total 569.94 last 22.39) - open/close 11319 (time 1786173177.22 total 579.97 last 18.26) - open/close 11580 (time 1786173187.26 total 590.01 last 25.99) - open/close 11726 (time 1786173198.56 total 601.31 last 12.92) - open/close 11964 (time 1786173208.58 total 611.33 last 23.76) - open/close 12128 (time 1786173218.58 total 621.33 last 16.39) - open/close 12412 (time 1786173229.25 total 631.99 last 26.63) - open/close 12640 (time 1786173239.26 total 642.01 last 22.76) - open/close 12738 (time 1786173249.65 total 652.40 last 9.43) - open/close 12982 (time 1786173260.44 total 663.19 last 22.61) - open/close 13178 (time 1786173270.44 total 673.19 last 19.59) - open/close 13394 (time 1786173281.49 total 684.24 last 19.56) - open/close 13571 (time 1786173292.45 total 695.20 last 16.14) - open/close 13794 (time 1786173303.14 total 705.89 last 20.87) - open/close 13987 (time 1786173313.15 total 715.90 last 19.28) - open/close 14130 (time 1786173323.42 total 726.17 last 13.92) - open/close 14380 (time 1786173333.93 total 736.68 last 23.79) - open/close 14616 (time 1786173344.68 total 747.43 last 21.95) - open/close 14778 (time 1786173354.83 total 757.58 last 15.96) - open/close 14899 (time 1786173365.42 total 768.17 last 11.43) - open/close 15065 (time 1786173376.09 total 778.84 last 15.56) - open/close 15250 (time 1786173386.73 total 789.48 last 17.39) - open/close 15486 (time 1786173397.66 total 800.41 last 21.58) - open/close 15677 (time 1786173408.41 total 811.15 last 17.78) - open/close 15863 (time 1786173418.41 total 821.16 last 18.60) - open/close 16108 (time 1786173428.42 total 831.17 last 24.47) - open/close 16345 (time 1786173438.45 total 841.20 last 23.62) - open/close 16460 (time 1786173449.33 total 852.08 last 10.57) - open/close 16586 (time 1786173460.30 total 863.05 last 11.49) - open/close 16839 (time 1786173470.93 total 873.67 last 23.82) - open/close 17050 (time 1786173480.94 total 883.68 last 21.08) - open/close 17319 (time 1786173491.93 total 894.68 last 24.48) - open/close 17539 (time 1786173502.92 total 905.67 last 20.01) - open/close 17783 (time 1786173513.90 total 916.65 last 22.22) - open/close 17981 (time 1786173525.72 total 928.47 last 16.75) - open/close 18198 (time 1786173535.73 total 938.48 last 21.67) - open/close 18389 (time 1786173546.10 total 948.85 last 18.43) - open/close 18658 (time 1786173556.10 total 958.85 last 26.89) - open/close 18846 (time 1786173566.74 total 969.49 last 17.67) - open/close 19041 (time 1786173576.75 total 979.49 last 19.49) - open/close 19480 (time 1786173586.75 total 989.50 last 43.89) - open/close 20000 (time 1786173592.76 total 995.50 last 86.56) - open/close 20877 (time 1786173602.77 total 1005.51 last 87.61) - open/close 21720 (time 1786173612.77 total 1015.52 last 84.28) - open/close 22575 (time 1786173622.78 total 1025.52 last 85.43) - open/close 23439 (time 1786173632.78 total 1035.53 last 86.35) - open/close 24316 (time 1786173642.79 total 1045.54 last 87.63) - open/close 25236 (time 1786173652.79 total 1055.54 last 91.96) - open/close 26126 (time 1786173662.80 total 1065.55 last 88.96) - open/close 26912 (time 1786173672.80 total 1075.55 last 78.57) - open/close 27703 (time 1786173682.81 total 1085.56 last 79.06) - open/close 28541 (time 1786173692.82 total 1095.57 last 83.73) - open/close 29407 (time 1786173702.82 total 1105.57 last 86.57) - open/close 30000 (time 1786173709.21 total 1111.96 last 92.76) - open/close 30884 (time 1786173719.22 total 1121.96 last 88.38) - open/close 31807 (time 1786173729.22 total 1131.97 last 92.22) - open/close 32631 (time 1786173739.22 total 1141.97 last 82.40) - open/close 33526 (time 1786173749.23 total 1151.97 last 89.49) - open/close 34288 (time 1786173759.24 total 1161.98 last 76.12) - open/close 35099 (time 1786173769.24 total 1171.99 last 81.06) - open/close 35929 (time 1786173779.25 total 1182.00 last 82.92) - open/close 36673 (time 1786173789.26 total 1192.01 last 74.34) - open/close 37398 (time 1786173799.27 total 1202.02 last 72.43) - open/close 38281 (time 1786173809.28 total 1212.03 last 88.22) - open/close 39005 (time 1786173819.28 total 1222.03 last 72.35) - open/close 39772 (time 1786173829.29 total 1232.04 last 76.66) - open/close 40000 (time 1786173832.64 total 1235.39 last 68.07) - open/close 40781 (time 1786173842.80 total 1245.55 last 76.84) - open/close 41571 (time 1786173852.86 total 1255.61 last 78.55) - open/close 42333 (time 1786173862.87 total 1265.61 last 76.15) - open/close 43127 (time 1786173872.87 total 1275.62 last 79.34) - open/close 43887 (time 1786173882.88 total 1285.63 last 75.93) - open/close 44756 (time 1786173892.89 total 1295.64 last 86.84) - open/close 45632 (time 1786173902.90 total 1305.65 last 87.50) - open/close 46433 (time 1786173912.91 total 1315.65 last 80.07) - open/close 47226 (time 1786173922.92 total 1325.67 last 79.17) - open/close 47992 (time 1786173932.93 total 1335.68 last 76.51) - open/close 48729 (time 1786173942.94 total 1345.68 last 73.68) - open/close 49538 (time 1786173953.39 total 1356.14 last 77.39) total: 50000 open/close in 1361.71 seconds: 36.72 ops/second - unlinked 0 (time 1786173964 ; total 0 ; last 0) - unlinked 10000 (time 1786174094 ; total 130 ; last 130) - unlinked 20000 (time 1786174235 ; total 271 ; last 141) - unlinked 30000 (time 1786174376 ; total 412 ; last 141) - unlinked 40000 (time 1786174520 ; total 556 ; last 144) total: 50000 unlinks in 694 seconds: 72.046112 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2