== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:43:37 (1787795017) start mds service on oleg228-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions oleg228-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg228-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg228-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 161 (time 1787795109.20 total 10.28 last 15.67) - open/close 354 (time 1787795119.22 total 20.30 last 19.26) - open/close 558 (time 1787795129.22 total 30.31 last 20.39) - open/close 786 (time 1787795139.85 total 40.94 last 21.45) - open/close 1028 (time 1787795149.87 total 50.95 last 24.17) - open/close 1258 (time 1787795160.00 total 61.08 last 22.71) - open/close 1409 (time 1787795171.98 total 73.07 last 12.60) - open/close 1570 (time 1787795183.63 total 84.71 last 13.83) - open/close 1758 (time 1787795194.53 total 95.61 last 17.25) - open/close 1979 (time 1787795204.55 total 105.64 last 22.04) - open/close 2092 (time 1787795215.24 total 116.33 last 10.57) - open/close 2150 (time 1787795225.95 total 127.03 last 5.42) - open/close 2301 (time 1787795235.95 total 137.03 last 15.10) - open/close 2487 (time 1787795246.60 total 147.68 last 17.46) - open/close 2762 (time 1787795256.61 total 157.69 last 27.48) - open/close 2911 (time 1787795266.64 total 167.72 last 14.86) - open/close 3177 (time 1787795277.52 total 178.60 last 24.45) - open/close 3325 (time 1787795289.11 total 190.19 last 12.77) - open/close 3603 (time 1787795299.36 total 200.44 last 27.13) - open/close 3842 (time 1787795310.27 total 211.35 last 21.90) - open/close 4056 (time 1787795321.28 total 222.36 last 19.43) - open/close 4178 (time 1787795331.30 total 232.38 last 12.18) - open/close 4314 (time 1787795341.30 total 242.38 last 13.59) - open/close 4539 (time 1787795351.95 total 253.03 last 21.14) - open/close 4758 (time 1787795364.81 total 265.89 last 17.03) - open/close 5011 (time 1787795375.58 total 276.66 last 23.48) - open/close 5264 (time 1787795385.58 total 286.66 last 25.29) - open/close 5447 (time 1787795395.60 total 296.68 last 18.27) - open/close 5662 (time 1787795406.65 total 307.73 last 19.45) - open/close 5826 (time 1787795417.03 total 318.11 last 15.80) - open/close 6024 (time 1787795427.14 total 328.22 last 19.59) - open/close 6228 (time 1787795439.38 total 340.46 last 16.66) - open/close 6403 (time 1787795449.83 total 350.91 last 16.75) - open/close 6663 (time 1787795459.84 total 360.92 last 25.97) - open/close 6786 (time 1787795469.86 total 370.94 last 12.27) - open/close 7027 (time 1787795480.85 total 381.93 last 21.94) - open/close 7263 (time 1787795490.86 total 391.94 last 23.57) - open/close 7441 (time 1787795501.38 total 402.46 last 16.92) - open/close 7589 (time 1787795513.41 total 414.49 last 12.30) - open/close 7896 (time 1787795524.29 total 425.38 last 28.20) - open/close 8045 (time 1787795534.31 total 435.40 last 14.87) - open/close 8202 (time 1787795545.03 total 446.12 last 14.65) - open/close 8385 (time 1787795555.98 total 457.07 last 16.71) - open/close 8604 (time 1787795566.23 total 467.31 last 21.37) - open/close 8857 (time 1787795576.25 total 477.33 last 25.25) - open/close 9132 (time 1787795586.25 total 487.33 last 27.50) - open/close 9415 (time 1787795596.26 total 497.34 last 28.27) - open/close 9666 (time 1787795607.03 total 508.11 last 23.30) - open/close 9907 (time 1787795617.78 total 518.86 last 22.42) - open/close 10000 (time 1787795625.83 total 526.91 last 11.56) - open/close 10250 (time 1787795635.83 total 536.91 last 25.00) - open/close 10381 (time 1787795645.84 total 546.92 last 13.09) - open/close 10471 (time 1787795655.85 total 556.93 last 8.99) - open/close 10665 (time 1787795666.88 total 567.96 last 17.58) - open/close 10868 (time 1787795676.91 total 577.99 last 20.24) - open/close 11067 (time 1787795686.93 total 588.01 last 19.86) - open/close 11133 (time 1787795696.94 total 598.02 last 6.60) - open/close 11188 (time 1787795706.94 total 608.02 last 5.50) - open/close 11369 (time 1787795716.95 total 618.03 last 18.08) - open/close 11492 (time 1787795728.46 total 629.55 last 10.69) - open/close 11754 (time 1787795738.50 total 639.58 last 26.12) - open/close 12015 (time 1787795749.01 total 650.09 last 24.83) - open/close 12264 (time 1787795759.04 total 660.12 last 24.82) - open/close 12523 (time 1787795769.95 total 671.04 last 23.73) - open/close 12750 (time 1787795780.96 total 682.04 last 20.62) - open/close 12896 (time 1787795790.99 total 692.08 last 14.56) - open/close 13146 (time 1787795802.02 total 703.10 last 22.68) - open/close 13352 (time 1787795813.02 total 714.10 last 18.73) - open/close 13585 (time 1787795823.03 total 724.11 last 23.27) - open/close 13738 (time 1787795833.04 total 734.12 last 15.29) - open/close 13992 (time 1787795843.05 total 744.13 last 25.37) - open/close 14263 (time 1787795853.89 total 754.97 last 24.98) - open/close 14481 (time 1787795866.57 total 767.65 last 17.20) - open/close 14753 (time 1787795877.25 total 778.33 last 25.46) - open/close 14997 (time 1787795888.20 total 789.28 last 22.28) - open/close 15219 (time 1787795898.50 total 799.59 last 21.55) - open/close 15469 (time 1787795908.52 total 809.60 last 24.97) - open/close 15748 (time 1787795919.51 total 820.60 last 25.37) - open/close 15960 (time 1787795930.52 total 831.60 last 19.27) - open/close 16173 (time 1787795941.38 total 842.47 last 19.60) - open/close 16337 (time 1787795951.39 total 852.47 last 16.39) - open/close 16464 (time 1787795964.43 total 865.51 last 9.74) - open/close 16649 (time 1787795975.30 total 876.38 last 17.03) - open/close 16901 (time 1787795985.31 total 886.40 last 25.16) - open/close 17033 (time 1787795996.42 total 897.50 last 11.88) - open/close 17235 (time 1787796006.58 total 907.66 last 19.88) - open/close 17416 (time 1787796016.60 total 917.68 last 18.07) - open/close 17613 (time 1787796026.76 total 927.84 last 19.39) - open/close 17784 (time 1787796037.05 total 938.13 last 16.62) - open/close 18031 (time 1787796047.06 total 948.15 last 24.66) - open/close 18242 (time 1787796057.87 total 958.96 last 19.52) - open/close 18491 (time 1787796068.25 total 969.33 last 23.99) - open/close 18733 (time 1787796079.11 total 980.19 last 22.29) - open/close 19412 (time 1787796089.14 total 990.22 last 67.73) - open/close 20000 (time 1787796096.96 total 998.04 last 75.18) - open/close 20736 (time 1787796106.96 total 1008.05 last 73.56) - open/close 21537 (time 1787796116.97 total 1018.05 last 80.06) - open/close 22370 (time 1787796126.99 total 1028.07 last 83.17) - open/close 23126 (time 1787796136.99 total 1038.07 last 75.60) - open/close 23619 (time 1787796146.99 total 1048.07 last 49.27) - open/close 24256 (time 1787796157.00 total 1058.08 last 63.67) - open/close 24999 (time 1787796167.00 total 1068.08 last 74.27) - open/close 25818 (time 1787796177.00 total 1078.08 last 81.89) - open/close 26597 (time 1787796187.02 total 1088.10 last 77.76) - open/close 27364 (time 1787796197.04 total 1098.12 last 76.55) - open/close 28042 (time 1787796207.05 total 1108.13 last 67.73) - open/close 28675 (time 1787796217.06 total 1118.14 last 63.24) - open/close 29350 (time 1787796227.07 total 1128.15 last 67.46) - open/close 30000 (time 1787796235.94 total 1137.02 last 73.25) - open/close 30733 (time 1787796245.94 total 1147.03 last 73.26) - open/close 31550 (time 1787796255.95 total 1157.03 last 81.64) - open/close 32378 (time 1787796265.96 total 1167.04 last 82.72) - open/close 33156 (time 1787796275.98 total 1177.06 last 77.68) - open/close 34014 (time 1787796285.98 total 1187.07 last 85.73) - open/close 34714 (time 1787796295.98 total 1197.07 last 70.00) - open/close 35352 (time 1787796305.99 total 1207.07 last 63.78) - open/close 35976 (time 1787796315.99 total 1217.07 last 62.38) - open/close 36684 (time 1787796326.00 total 1227.08 last 70.75) - open/close 37542 (time 1787796336.01 total 1237.09 last 85.68) - open/close 38321 (time 1787796346.02 total 1247.10 last 77.87) - open/close 39117 (time 1787796356.03 total 1257.11 last 79.45) - open/close 39795 (time 1787796366.04 total 1267.12 last 67.77) - open/close 40000 (time 1787796368.66 total 1269.74 last 78.15) - open/close 40790 (time 1787796378.69 total 1279.77 last 78.76) - open/close 41525 (time 1787796388.70 total 1289.78 last 73.45) - open/close 42335 (time 1787796398.70 total 1299.79 last 80.95) - open/close 43098 (time 1787796408.72 total 1309.80 last 76.20) - open/close 43800 (time 1787796418.73 total 1319.81 last 70.10) - open/close 44446 (time 1787796428.74 total 1329.82 last 64.55) - open/close 45103 (time 1787796438.75 total 1339.83 last 65.61) - open/close 45742 (time 1787796448.77 total 1349.85 last 63.81) - open/close 46470 (time 1787796458.77 total 1359.85 last 72.76) - open/close 47191 (time 1787796468.82 total 1369.90 last 71.78) - open/close 47914 (time 1787796478.85 total 1379.94 last 72.03) - open/close 48650 (time 1787796488.87 total 1389.95 last 73.49) - open/close 49305 (time 1787796498.88 total 1399.96 last 65.41) total: 50000 open/close in 1407.81 seconds: 35.52 ops/second - unlinked 0 (time 1787796511 ; total 0 ; last 0) - unlinked 10000 (time 1787796645 ; total 134 ; last 134) - unlinked 20000 (time 1787796785 ; total 274 ; last 140) - unlinked 30000 (time 1787796922 ; total 411 ; last 137) - unlinked 40000 (time 1787797070 ; total 559 ; last 148) total: 50000 unlinks in 696 seconds: 71.839081 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: modules unloaded. pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2