== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:22:01 (1787736121) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 241 (time 1787736205.54 total 10.00 last 24.09) - open/close 441 (time 1787736215.54 total 20.01 last 19.99) - open/close 670 (time 1787736225.55 total 30.01 last 22.88) - open/close 843 (time 1787736235.78 total 40.24 last 16.91) - open/close 1052 (time 1787736246.14 total 50.60 last 20.18) - open/close 1258 (time 1787736256.58 total 61.04 last 19.73) - open/close 1501 (time 1787736267.25 total 71.71 last 22.77) - open/close 1726 (time 1787736277.65 total 82.12 last 21.62) - open/close 1963 (time 1787736287.87 total 92.33 last 23.20) - open/close 2204 (time 1787736297.88 total 102.34 last 24.07) - open/close 2504 (time 1787736308.88 total 113.34 last 27.28) - open/close 2728 (time 1787736318.89 total 123.36 last 22.37) - open/close 2871 (time 1787736328.90 total 133.36 last 14.30) - open/close 3044 (time 1787736339.14 total 143.60 last 16.89) - open/close 3276 (time 1787736349.84 total 154.30 last 21.69) - open/close 3612 (time 1787736360.32 total 164.79 last 32.03) - open/close 3843 (time 1787736370.36 total 174.82 last 23.03) - open/close 4063 (time 1787736380.49 total 184.95 last 21.71) - open/close 4297 (time 1787736390.61 total 195.07 last 23.12) - open/close 4538 (time 1787736400.62 total 205.08 last 24.07) - open/close 4768 (time 1787736410.63 total 215.09 last 22.98) - open/close 4946 (time 1787736421.49 total 225.96 last 16.38) - open/close 5185 (time 1787736432.01 total 236.48 last 22.72) - open/close 5414 (time 1787736442.45 total 246.91 last 21.95) - open/close 5590 (time 1787736452.46 total 256.93 last 17.57) - open/close 5788 (time 1787736462.71 total 267.18 last 19.31) - open/close 6015 (time 1787736473.53 total 278.00 last 20.98) - open/close 6222 (time 1787736483.54 total 288.01 last 20.68) - open/close 6460 (time 1787736493.55 total 298.01 last 23.79) - open/close 6709 (time 1787736503.93 total 308.39 last 24.00) - open/close 6947 (time 1787736514.34 total 318.80 last 22.86) - open/close 7195 (time 1787736524.35 total 328.81 last 24.78) - open/close 7356 (time 1787736535.15 total 339.61 last 14.90) - open/close 7567 (time 1787736545.15 total 349.61 last 21.10) - open/close 7709 (time 1787736555.21 total 359.67 last 14.12) - open/close 7939 (time 1787736566.15 total 370.61 last 21.02) - open/close 8180 (time 1787736576.45 total 380.91 last 23.40) - open/close 8389 (time 1787736586.82 total 391.29 last 20.15) - open/close 8682 (time 1787736597.12 total 401.58 last 28.47) - open/close 8946 (time 1787736607.42 total 411.89 last 25.61) - open/close 9195 (time 1787736617.43 total 421.90 last 24.88) - open/close 9410 (time 1787736627.44 total 431.90 last 21.49) - open/close 9627 (time 1787736638.28 total 442.75 last 20.00) - open/close 9884 (time 1787736649.16 total 453.63 last 23.62) - open/close 10000 (time 1787736653.25 total 457.71 last 28.42) - open/close 10279 (time 1787736663.26 total 467.72 last 27.88) - open/close 10471 (time 1787736674.18 total 478.64 last 17.57) - open/close 10712 (time 1787736684.61 total 489.07 last 23.11) - open/close 10932 (time 1787736695.67 total 500.13 last 19.89) - open/close 11157 (time 1787736705.95 total 510.41 last 21.90) - open/close 11428 (time 1787736716.81 total 521.27 last 24.95) - open/close 11652 (time 1787736726.91 total 531.37 last 22.17) - open/close 11843 (time 1787736737.60 total 542.06 last 17.87) - open/close 12057 (time 1787736749.72 total 554.18 last 17.66) - open/close 12289 (time 1787736760.19 total 564.65 last 22.16) - open/close 12521 (time 1787736770.32 total 574.78 last 22.91) - open/close 12780 (time 1787736781.12 total 585.58 last 23.98) - open/close 13011 (time 1787736791.18 total 595.64 last 22.96) - open/close 13276 (time 1787736802.06 total 606.52 last 24.36) - open/close 13466 (time 1787736812.09 total 616.55 last 18.94) - open/close 13680 (time 1787736822.10 total 626.56 last 21.38) - open/close 14116 (time 1787736832.11 total 636.57 last 43.56) - open/close 14324 (time 1787736842.62 total 647.08 last 19.79) - open/close 14469 (time 1787736853.52 total 657.98 last 13.30) - open/close 14693 (time 1787736863.52 total 667.99 last 22.39) - open/close 14867 (time 1787736874.59 total 679.05 last 15.73) - open/close 14982 (time 1787736884.60 total 689.06 last 11.48) - open/close 15051 (time 1787736894.60 total 699.07 last 6.90) - open/close 15146 (time 1787736905.21 total 709.68 last 8.95) - open/close 15324 (time 1787736915.24 total 719.71 last 17.75) - open/close 15490 (time 1787736925.26 total 729.73 last 16.57) - open/close 15594 (time 1787736935.30 total 739.76 last 10.36) - open/close 15688 (time 1787736945.31 total 749.77 last 9.39) - open/close 15829 (time 1787736955.42 total 759.89 last 13.94) - open/close 15997 (time 1787736965.67 total 770.13 last 16.40) - open/close 16110 (time 1787736977.61 total 782.07 last 9.46) - open/close 16223 (time 1787736987.96 total 792.42 last 10.92) - open/close 16376 (time 1787736998.54 total 803.00 last 14.46) - open/close 16470 (time 1787737008.55 total 813.01 last 9.39) - open/close 16619 (time 1787737019.34 total 823.80 last 13.81) - open/close 16726 (time 1787737030.22 total 834.68 last 9.83) - open/close 16851 (time 1787737040.39 total 844.85 last 12.29) - open/close 17062 (time 1787737050.58 total 855.05 last 20.70) - open/close 17223 (time 1787737061.27 total 865.73 last 15.07) - open/close 17401 (time 1787737073.73 total 878.20 last 14.28) - open/close 17577 (time 1787737083.74 total 888.20 last 17.59) - open/close 17770 (time 1787737094.10 total 898.56 last 18.64) - open/close 17856 (time 1787737104.55 total 909.01 last 8.23) - open/close 18028 (time 1787737114.56 total 919.02 last 17.18) - open/close 18198 (time 1787737126.10 total 930.56 last 14.74) - open/close 18856 (time 1787737136.10 total 940.56 last 65.78) - open/close 19552 (time 1787737146.10 total 950.56 last 69.60) - open/close 20000 (time 1787737152.81 total 957.28 last 66.72) - open/close 20778 (time 1787737162.82 total 967.28 last 77.78) - open/close 21584 (time 1787737172.82 total 977.29 last 80.56) - open/close 22372 (time 1787737182.83 total 987.29 last 78.73) - open/close 23069 (time 1787737192.85 total 997.32 last 69.54) - open/close 23779 (time 1787737202.85 total 1007.32 last 71.00) - open/close 24470 (time 1787737212.85 total 1017.32 last 69.10) - open/close 25170 (time 1787737222.86 total 1027.33 last 69.95) - open/close 25926 (time 1787737232.86 total 1037.33 last 75.58) - open/close 26656 (time 1787737242.87 total 1047.34 last 72.93) - open/close 27342 (time 1787737252.87 total 1057.34 last 68.60) - open/close 28015 (time 1787737262.88 total 1067.35 last 67.24) - open/close 28663 (time 1787737272.90 total 1077.36 last 64.69) - open/close 29424 (time 1787737282.90 total 1087.37 last 76.07) - open/close 30000 (time 1787737290.66 total 1095.12 last 74.28) - open/close 30725 (time 1787737300.66 total 1105.13 last 72.46) - open/close 31482 (time 1787737310.67 total 1115.14 last 75.62) - open/close 32118 (time 1787737320.69 total 1125.15 last 63.51) - open/close 32709 (time 1787737330.69 total 1135.15 last 59.09) - open/close 33296 (time 1787737340.70 total 1145.17 last 58.62) - open/close 33967 (time 1787737350.70 total 1155.17 last 67.10) - open/close 34630 (time 1787737360.73 total 1165.19 last 66.16) - open/close 35281 (time 1787737370.73 total 1175.19 last 65.10) - open/close 36022 (time 1787737380.74 total 1185.20 last 74.00) - open/close 36743 (time 1787737390.74 total 1195.21 last 72.07) - open/close 37488 (time 1787737400.75 total 1205.22 last 74.42) - open/close 38244 (time 1787737410.77 total 1215.23 last 75.50) - open/close 38934 (time 1787737420.77 total 1225.24 last 68.96) - open/close 39663 (time 1787737430.78 total 1235.25 last 72.83) - open/close 40000 (time 1787737435.89 total 1240.35 last 65.99) - open/close 40670 (time 1787737445.89 total 1250.36 last 66.98) - open/close 41405 (time 1787737455.90 total 1260.36 last 73.45) - open/close 42097 (time 1787737465.91 total 1270.37 last 69.15) - open/close 42804 (time 1787737475.92 total 1280.39 last 70.59) - open/close 43566 (time 1787737485.93 total 1290.39 last 76.15) - open/close 44173 (time 1787737495.94 total 1300.40 last 60.64) - open/close 44867 (time 1787737505.94 total 1310.41 last 69.39) - open/close 45514 (time 1787737515.95 total 1320.41 last 64.67) - open/close 46185 (time 1787737525.95 total 1330.42 last 67.05) - open/close 46877 (time 1787737535.97 total 1340.44 last 69.06) - open/close 47564 (time 1787737545.98 total 1350.44 last 68.68) - open/close 48269 (time 1787737555.98 total 1360.45 last 70.45) - open/close 48928 (time 1787737565.98 total 1370.45 last 65.90) - open/close 49605 (time 1787737576.00 total 1380.46 last 67.62) total: 50000 open/close in 1386.12 seconds: 36.07 ops/second - unlinked 0 (time 1787737585 ; total 0 ; last 0) - unlinked 10000 (time 1787737738 ; total 153 ; last 153) - unlinked 20000 (time 1787737891 ; total 306 ; last 153) - unlinked 30000 (time 1787738040 ; total 455 ; last 149) - unlinked 40000 (time 1787738189 ; total 604 ; last 149) total: 50000 unlinks in 740 seconds: 67.567566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local oleg111-server: modules unloaded. pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2