== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:26:47 (1787826407) start mds service on oleg645-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 233 (time 1787826496.62 total 10.01 last 23.27) - open/close 329 (time 1787826506.87 total 20.26 last 9.37) - open/close 578 (time 1787826516.88 total 30.28 last 24.86) - open/close 765 (time 1787826527.27 total 40.67 last 18.00) - open/close 950 (time 1787826537.57 total 50.96 last 17.97) - open/close 1208 (time 1787826547.59 total 60.98 last 25.75) - open/close 1392 (time 1787826557.59 total 70.99 last 18.39) - open/close 1520 (time 1787826567.98 total 81.38 last 12.32) - open/close 1752 (time 1787826578.91 total 92.30 last 21.23) - open/close 1848 (time 1787826589.48 total 102.87 last 9.09) - open/close 1999 (time 1787826600.03 total 113.42 last 14.31) - open/close 2206 (time 1787826610.21 total 123.61 last 20.32) - open/close 2444 (time 1787826621.80 total 135.19 last 20.55) - open/close 2667 (time 1787826631.82 total 145.22 last 22.23) - open/close 2869 (time 1787826642.85 total 156.25 last 18.32) - open/close 3043 (time 1787826652.86 total 166.25 last 17.39) - open/close 3288 (time 1787826662.87 total 176.26 last 24.48) - open/close 3429 (time 1787826673.32 total 186.71 last 13.49) - open/close 3707 (time 1787826683.61 total 197.01 last 27.01) - open/close 3922 (time 1787826694.05 total 207.45 last 20.60) - open/close 4147 (time 1787826704.07 total 217.46 last 22.46) - open/close 4375 (time 1787826714.22 total 227.62 last 22.45) - open/close 4619 (time 1787826725.73 total 239.12 last 21.21) - open/close 4846 (time 1787826736.22 total 249.61 last 21.65) - open/close 5079 (time 1787826746.58 total 259.98 last 22.48) - open/close 5326 (time 1787826756.70 total 270.09 last 24.42) - open/close 5569 (time 1787826766.71 total 280.10 last 24.28) - open/close 5821 (time 1787826776.72 total 290.12 last 25.16) - open/close 6025 (time 1787826787.49 total 300.88 last 18.95) - open/close 6211 (time 1787826798.37 total 311.77 last 17.09) - open/close 6465 (time 1787826808.38 total 321.77 last 25.39) - open/close 6686 (time 1787826818.98 total 332.38 last 20.83) - open/close 6830 (time 1787826829.02 total 342.41 last 14.35) - open/close 7054 (time 1787826839.14 total 352.53 last 22.13) - open/close 7297 (time 1787826849.15 total 362.55 last 24.27) - open/close 7546 (time 1787826859.61 total 373.01 last 23.80) - open/close 7833 (time 1787826869.61 total 383.01 last 28.70) - open/close 7992 (time 1787826879.62 total 393.01 last 15.89) - open/close 8167 (time 1787826890.35 total 403.74 last 16.31) - open/close 8335 (time 1787826900.36 total 413.75 last 16.79) - open/close 8599 (time 1787826911.15 total 424.54 last 24.46) - open/close 8826 (time 1787826921.19 total 434.59 last 22.60) - open/close 9085 (time 1787826931.62 total 445.01 last 24.85) - open/close 9290 (time 1787826942.63 total 456.02 last 18.62) - open/close 9544 (time 1787826953.38 total 466.78 last 23.62) - open/close 9722 (time 1787826963.39 total 476.78 last 17.78) - open/close 9949 (time 1787826973.41 total 486.81 last 22.65) - open/close 10000 (time 1787826975.56 total 488.95 last 23.76) - open/close 10215 (time 1787826985.56 total 498.95 last 21.50) - open/close 10420 (time 1787826995.69 total 509.09 last 20.23) - open/close 10681 (time 1787827005.70 total 519.10 last 26.07) - open/close 10888 (time 1787827015.71 total 529.11 last 20.68) - open/close 11127 (time 1787827026.27 total 539.67 last 22.63) - open/close 11357 (time 1787827036.70 total 550.09 last 22.06) - open/close 11592 (time 1787827047.00 total 560.40 last 22.81) - open/close 11811 (time 1787827058.02 total 571.41 last 19.88) - open/close 11904 (time 1787827068.76 total 582.15 last 8.66) - open/close 12082 (time 1787827078.79 total 592.19 last 17.74) - open/close 12293 (time 1787827088.92 total 602.31 last 20.84) - open/close 12537 (time 1787827099.35 total 612.75 last 23.38) - open/close 12768 (time 1787827109.47 total 622.86 last 22.84) - open/close 13025 (time 1787827119.90 total 633.29 last 24.65) - open/close 13251 (time 1787827130.33 total 643.73 last 21.65) - open/close 13442 (time 1787827140.91 total 654.30 last 18.06) - open/close 13661 (time 1787827151.20 total 664.59 last 21.28) - open/close 13821 (time 1787827161.76 total 675.15 last 15.16) - open/close 14059 (time 1787827171.77 total 685.17 last 23.75) - open/close 14205 (time 1787827184.12 total 697.51 last 11.83) - open/close 14368 (time 1787827197.05 total 710.45 last 12.60) - open/close 14555 (time 1787827207.06 total 720.45 last 18.69) - open/close 14767 (time 1787827217.07 total 730.47 last 21.17) - open/close 15005 (time 1787827228.08 total 741.47 last 21.62) - open/close 15219 (time 1787827238.42 total 751.82 last 20.69) - open/close 15426 (time 1787827248.43 total 761.82 last 20.70) - open/close 15639 (time 1787827258.44 total 771.83 last 21.27) - open/close 15840 (time 1787827268.44 total 781.83 last 20.10) - open/close 15940 (time 1787827278.81 total 792.21 last 9.64) - open/close 16115 (time 1787827289.58 total 802.98 last 16.25) - open/close 16283 (time 1787827300.51 total 813.91 last 15.37) - open/close 16388 (time 1787827310.54 total 823.93 last 10.47) - open/close 16676 (time 1787827321.14 total 834.53 last 27.17) - open/close 16905 (time 1787827331.36 total 844.75 last 22.40) - open/close 17193 (time 1787827341.37 total 854.77 last 28.76) - open/close 17326 (time 1787827351.38 total 864.77 last 13.29) - open/close 17574 (time 1787827361.38 total 874.78 last 24.79) - open/close 17767 (time 1787827371.40 total 884.79 last 19.27) - open/close 18023 (time 1787827382.02 total 895.41 last 24.11) - open/close 18197 (time 1787827392.66 total 906.05 last 16.35) - open/close 18422 (time 1787827403.95 total 917.35 last 19.92) - open/close 18620 (time 1787827414.87 total 928.26 last 18.14) - open/close 18898 (time 1787827424.88 total 938.28 last 27.76) - open/close 19505 (time 1787827434.89 total 948.28 last 60.68) - open/close 20000 (time 1787827442.12 total 955.51 last 68.43) - open/close 20699 (time 1787827452.12 total 965.52 last 69.88) - open/close 21390 (time 1787827462.14 total 975.53 last 69.02) - open/close 22117 (time 1787827472.14 total 985.53 last 72.69) - open/close 22853 (time 1787827482.14 total 995.54 last 73.57) - open/close 23576 (time 1787827492.15 total 1005.55 last 72.21) - open/close 24258 (time 1787827502.16 total 1015.56 last 68.13) - open/close 24932 (time 1787827512.18 total 1025.57 last 67.30) - open/close 25610 (time 1787827522.21 total 1035.60 last 67.61) - open/close 26262 (time 1787827532.24 total 1045.63 last 65.01) - open/close 26944 (time 1787827542.24 total 1055.63 last 68.19) - open/close 27634 (time 1787827552.24 total 1065.64 last 68.98) - open/close 28325 (time 1787827562.25 total 1075.64 last 69.05) - open/close 28981 (time 1787827572.26 total 1085.66 last 65.50) - open/close 29656 (time 1787827582.26 total 1095.66 last 67.50) - open/close 30000 (time 1787827587.44 total 1100.83 last 66.48) - open/close 30668 (time 1787827597.46 total 1110.85 last 66.67) - open/close 31314 (time 1787827607.47 total 1120.87 last 64.49) - open/close 31907 (time 1787827617.48 total 1130.87 last 59.26) - open/close 32495 (time 1787827627.48 total 1140.88 last 58.80) - open/close 33172 (time 1787827637.49 total 1150.89 last 67.63) - open/close 33825 (time 1787827647.50 total 1160.89 last 65.26) - open/close 34488 (time 1787827657.50 total 1170.90 last 66.27) - open/close 35179 (time 1787827667.51 total 1180.91 last 69.02) - open/close 35858 (time 1787827677.52 total 1190.92 last 67.83) - open/close 36583 (time 1787827687.53 total 1200.92 last 72.45) - open/close 37280 (time 1787827697.54 total 1210.93 last 69.66) - open/close 37926 (time 1787827707.54 total 1220.94 last 64.55) - open/close 38581 (time 1787827717.57 total 1230.97 last 65.31) - open/close 39250 (time 1787827727.58 total 1240.97 last 66.85) - open/close 39967 (time 1787827737.59 total 1250.98 last 71.64) - open/close 40678 (time 1787827747.60 total 1261.00 last 71.00) - open/close 41373 (time 1787827757.61 total 1271.00 last 69.44) - open/close 42029 (time 1787827767.64 total 1281.03 last 65.42) - open/close 42670 (time 1787827777.65 total 1291.04 last 64.04) - open/close 43231 (time 1787827787.66 total 1301.05 last 56.03) - open/close 43796 (time 1787827797.66 total 1311.05 last 56.49) - open/close 44389 (time 1787827807.66 total 1321.06 last 59.27) - open/close 44871 (time 1787827817.70 total 1331.09 last 48.03) - open/close 45525 (time 1787827827.70 total 1341.09 last 65.40) - open/close 46195 (time 1787827837.71 total 1351.10 last 66.96) - open/close 46884 (time 1787827847.71 total 1361.11 last 68.85) - open/close 47516 (time 1787827857.72 total 1371.11 last 63.18) - open/close 48084 (time 1787827867.73 total 1381.12 last 56.75) - open/close 48616 (time 1787827877.73 total 1391.13 last 53.16) - open/close 49245 (time 1787827887.74 total 1401.14 last 62.85) - open/close 49903 (time 1787827897.75 total 1411.15 last 65.72) total: 50000 open/close in 1412.16 seconds: 35.41 ops/second - unlinked 0 (time 1787827903 ; total 0 ; last 0) - unlinked 10000 (time 1787828052 ; total 149 ; last 149) - unlinked 20000 (time 1787828201 ; total 298 ; last 149) - unlinked 30000 (time 1787828351 ; total 448 ; last 150) - unlinked 40000 (time 1787828497 ; total 594 ; last 146) total: 50000 unlinks in 739 seconds: 67.658997 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2