== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:57:12 (1775869032) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 177 (time 1775869122.18 total 10.90 last 16.24) - open/close 380 (time 1775869132.55 total 21.27 last 19.58) - open/close 614 (time 1775869142.56 total 31.28 last 23.38) - open/close 778 (time 1775869153.09 total 41.81 last 15.57) - open/close 941 (time 1775869166.09 total 54.81 last 12.54) - open/close 1116 (time 1775869176.10 total 64.82 last 17.49) - open/close 1327 (time 1775869186.41 total 75.13 last 20.46) - open/close 1523 (time 1775869196.74 total 85.47 last 18.97) - open/close 1749 (time 1775869206.76 total 95.48 last 22.58) - open/close 1955 (time 1775869216.94 total 105.66 last 20.22) - open/close 2168 (time 1775869226.94 total 115.66 last 21.30) - open/close 2378 (time 1775869237.26 total 125.98 last 20.35) - open/close 2618 (time 1775869247.27 total 135.99 last 23.98) - open/close 2812 (time 1775869257.27 total 145.99 last 19.39) - open/close 3018 (time 1775869267.29 total 156.01 last 20.56) - open/close 3173 (time 1775869277.90 total 166.62 last 14.61) - open/close 3425 (time 1775869288.09 total 176.81 last 24.73) - open/close 3650 (time 1775869298.13 total 186.85 last 22.40) - open/close 3798 (time 1775869309.93 total 198.65 last 12.55) - open/close 4027 (time 1775869319.95 total 208.67 last 22.85) - open/close 4264 (time 1775869329.95 total 218.67 last 23.68) - open/close 4512 (time 1775869339.96 total 228.68 last 24.78) - open/close 4738 (time 1775869350.11 total 238.83 last 22.28) - open/close 4967 (time 1775869360.13 total 248.85 last 22.84) - open/close 5181 (time 1775869370.40 total 259.12 last 20.85) - open/close 5387 (time 1775869381.08 total 269.80 last 19.29) - open/close 5608 (time 1775869391.08 total 279.80 last 22.09) - open/close 5806 (time 1775869401.09 total 289.81 last 19.78) - open/close 6011 (time 1775869411.21 total 299.94 last 20.25) - open/close 6222 (time 1775869421.24 total 309.96 last 21.05) - open/close 6446 (time 1775869431.26 total 319.98 last 22.36) - open/close 6628 (time 1775869441.62 total 330.34 last 17.56) - open/close 6839 (time 1775869451.63 total 340.35 last 21.08) - open/close 7051 (time 1775869461.64 total 350.36 last 21.17) - open/close 7304 (time 1775869471.64 total 360.36 last 25.30) - open/close 7504 (time 1775869482.06 total 370.78 last 19.19) - open/close 7700 (time 1775869492.23 total 380.95 last 19.28) - open/close 7882 (time 1775869502.25 total 390.97 last 18.16) - open/close 8136 (time 1775869513.22 total 401.94 last 23.15) - open/close 8317 (time 1775869523.34 total 412.07 last 17.88) - open/close 8539 (time 1775869533.71 total 422.43 last 21.42) - open/close 8783 (time 1775869544.53 total 433.25 last 22.54) - open/close 8980 (time 1775869555.14 total 443.86 last 18.56) - open/close 9157 (time 1775869565.59 total 454.31 last 16.95) - open/close 9371 (time 1775869575.59 total 464.31 last 21.38) - open/close 9609 (time 1775869585.62 total 474.34 last 23.75) - open/close 9839 (time 1775869595.62 total 484.34 last 22.98) - open/close 10000 (time 1775869604.61 total 493.33 last 17.91) - open/close 10137 (time 1775869614.64 total 503.36 last 13.66) - open/close 10291 (time 1775869625.29 total 514.01 last 14.46) - open/close 10406 (time 1775869635.49 total 524.21 last 11.27) - open/close 10525 (time 1775869645.59 total 534.31 last 11.78) - open/close 10709 (time 1775869656.20 total 544.92 last 17.34) - open/close 10954 (time 1775869666.21 total 554.93 last 24.49) - open/close 11210 (time 1775869676.89 total 565.61 last 23.96) - open/close 11478 (time 1775869686.92 total 575.64 last 26.74) - open/close 11679 (time 1775869697.80 total 586.52 last 18.47) - open/close 11897 (time 1775869707.80 total 596.52 last 21.80) - open/close 12133 (time 1775869718.42 total 607.14 last 22.21) - open/close 12292 (time 1775869728.60 total 617.32 last 15.62) - open/close 12471 (time 1775869739.22 total 627.94 last 16.87) - open/close 12659 (time 1775869750.11 total 638.83 last 17.26) - open/close 12827 (time 1775869760.86 total 649.58 last 15.63) - open/close 13027 (time 1775869773.84 total 662.56 last 15.41) - open/close 13202 (time 1775869784.48 total 673.20 last 16.45) - open/close 13398 (time 1775869794.49 total 683.21 last 19.57) - open/close 13578 (time 1775869805.26 total 693.98 last 16.73) - open/close 13678 (time 1775869816.04 total 704.76 last 9.27) - open/close 13861 (time 1775869826.95 total 715.67 last 16.78) - open/close 14090 (time 1775869837.72 total 726.44 last 21.28) - open/close 14177 (time 1775869847.73 total 736.45 last 8.69) - open/close 14383 (time 1775869857.75 total 746.47 last 20.56) - open/close 14549 (time 1775869867.77 total 756.49 last 16.56) - open/close 14646 (time 1775869877.91 total 766.63 last 9.57) - open/close 14786 (time 1775869888.35 total 777.07 last 13.41) - open/close 15003 (time 1775869898.83 total 787.55 last 20.71) - open/close 15103 (time 1775869909.51 total 798.23 last 9.36) - open/close 15203 (time 1775869920.21 total 808.93 last 9.34) - open/close 15386 (time 1775869930.32 total 819.04 last 18.10) - open/close 15542 (time 1775869940.33 total 829.05 last 15.59) - open/close 15604 (time 1775869951.19 total 839.91 last 5.71) - open/close 15860 (time 1775869961.20 total 849.92 last 25.57) - open/close 16046 (time 1775869971.20 total 859.92 last 18.59) - open/close 16279 (time 1775869981.21 total 869.93 last 23.29) - open/close 16502 (time 1775869991.56 total 880.28 last 21.54) - open/close 16729 (time 1775870001.58 total 890.30 last 22.66) - open/close 16902 (time 1775870011.60 total 900.32 last 17.27) - open/close 17048 (time 1775870021.91 total 910.63 last 14.16) - open/close 17225 (time 1775870031.93 total 920.65 last 17.66) - open/close 17456 (time 1775870041.94 total 930.66 last 23.09) - open/close 17583 (time 1775870053.50 total 942.22 last 10.98) - open/close 17788 (time 1775870063.50 total 952.22 last 20.49) - open/close 18415 (time 1775870073.51 total 962.23 last 62.62) - open/close 19046 (time 1775870083.54 total 972.26 last 62.97) - open/close 19637 (time 1775870093.54 total 982.26 last 59.08) - open/close 20000 (time 1775870101.41 total 990.13 last 46.15) - open/close 20673 (time 1775870111.41 total 1000.13 last 67.26) - open/close 21315 (time 1775870121.42 total 1010.14 last 64.13) - open/close 21975 (time 1775870131.43 total 1020.15 last 65.94) - open/close 22622 (time 1775870141.43 total 1030.15 last 64.70) - open/close 23262 (time 1775870151.43 total 1040.15 last 64.00) - open/close 23869 (time 1775870161.44 total 1050.16 last 60.67) - open/close 24518 (time 1775870171.45 total 1060.17 last 64.85) - open/close 25224 (time 1775870181.46 total 1070.18 last 70.50) - open/close 25846 (time 1775870191.46 total 1080.18 last 62.20) - open/close 26366 (time 1775870201.47 total 1090.19 last 51.97) - open/close 26987 (time 1775870211.47 total 1100.19 last 62.05) - open/close 27562 (time 1775870221.49 total 1110.21 last 57.42) - open/close 28147 (time 1775870231.50 total 1120.22 last 58.43) - open/close 28687 (time 1775870241.51 total 1130.23 last 53.92) - open/close 29286 (time 1775870251.51 total 1140.23 last 59.90) - open/close 29859 (time 1775870261.52 total 1150.24 last 57.29) - open/close 30000 (time 1775870264.01 total 1152.73 last 56.56) - open/close 30591 (time 1775870274.02 total 1162.74 last 59.06) - open/close 31197 (time 1775870284.02 total 1172.74 last 60.56) - open/close 31762 (time 1775870294.04 total 1182.76 last 56.38) - open/close 32356 (time 1775870304.05 total 1192.77 last 59.35) - open/close 32955 (time 1775870314.06 total 1202.78 last 59.87) - open/close 33592 (time 1775870324.07 total 1212.79 last 63.62) - open/close 34199 (time 1775870334.09 total 1222.81 last 60.57) - open/close 34764 (time 1775870344.10 total 1232.82 last 56.45) - open/close 35388 (time 1775870354.10 total 1242.82 last 62.40) - open/close 35966 (time 1775870364.12 total 1252.84 last 57.69) - open/close 36521 (time 1775870374.13 total 1262.85 last 55.44) - open/close 37115 (time 1775870384.13 total 1272.85 last 59.39) - open/close 37692 (time 1775870394.15 total 1282.87 last 57.58) - open/close 38216 (time 1775870404.15 total 1292.87 last 52.37) - open/close 38779 (time 1775870414.16 total 1302.88 last 56.29) - open/close 39284 (time 1775870424.18 total 1312.90 last 50.38) - open/close 39833 (time 1775870434.19 total 1322.91 last 54.85) - open/close 40000 (time 1775870437.47 total 1326.19 last 50.90) - open/close 40559 (time 1775870447.48 total 1336.20 last 55.86) - open/close 41121 (time 1775870457.50 total 1346.22 last 56.06) - open/close 41681 (time 1775870467.52 total 1356.24 last 55.90) - open/close 42288 (time 1775870477.53 total 1366.25 last 60.65) - open/close 42894 (time 1775870487.53 total 1376.25 last 60.56) - open/close 43526 (time 1775870497.54 total 1386.26 last 63.18) - open/close 44150 (time 1775870507.54 total 1396.27 last 62.35) - open/close 44706 (time 1775870517.55 total 1406.27 last 55.59) - open/close 45260 (time 1775870527.56 total 1416.28 last 55.30) - open/close 45828 (time 1775870537.58 total 1426.30 last 56.73) - open/close 46417 (time 1775870547.62 total 1436.34 last 58.62) - open/close 46999 (time 1775870557.63 total 1446.35 last 58.16) - open/close 47585 (time 1775870567.64 total 1456.36 last 58.58) - open/close 48170 (time 1775870577.64 total 1466.36 last 58.49) - open/close 48684 (time 1775870587.65 total 1476.37 last 51.34) - open/close 49198 (time 1775870597.65 total 1486.37 last 51.39) - open/close 49690 (time 1775870607.65 total 1496.37 last 49.18) total: 50000 open/close in 1502.98 seconds: 33.27 ops/second - unlinked 0 (time 1775870619 ; total 0 ; last 0) - unlinked 10000 (time 1775870816 ; total 197 ; last 197) - unlinked 20000 (time 1775870998 ; total 379 ; last 182) - unlinked 30000 (time 1775871179 ; total 560 ; last 181) - unlinked 40000 (time 1775871359 ; total 740 ; last 180) total: 50000 unlinks in 916 seconds: 54.585152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: modules unloaded. pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2