== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:42:20 (1786423340) start mds service on oleg111-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing load_modules_local oleg111-server: Loading modules from /home/green/git/lustre-release/lustre oleg111-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg111-server: detected 4 online CPUs by sysfs oleg111-server: MODOPTS_LIBCFS= oleg111-server: Force libcfs to create 2 CPU partitions oleg111-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg111-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg111-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg111-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg111-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg111-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg111-server: oleg111-server.virtnet: executing set_default_debug -1 all pdsh@oleg111-client: oleg111-server: ssh exited with exit code 1 Started lustre-OST0000 oleg111-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg111-client.virtnet: -o user_xattr,flock 192.168.201.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 122 (time 1786423475.30 total 10.10 last 12.08) - open/close 170 (time 1786423485.78 total 20.58 last 4.58) - open/close 355 (time 1786423495.78 total 30.59 last 18.49) - open/close 497 (time 1786423505.93 total 40.73 last 13.99) - open/close 562 (time 1786423516.04 total 50.85 last 6.43) - open/close 687 (time 1786423526.73 total 61.54 last 11.69) - open/close 866 (time 1786423537.57 total 72.37 last 16.52) - open/close 954 (time 1786423547.57 total 82.37 last 8.80) - open/close 1134 (time 1786423558.09 total 92.90 last 17.10) - open/close 1186 (time 1786423568.84 total 103.64 last 4.84) - open/close 1398 (time 1786423578.84 total 113.65 last 21.19) - open/close 1518 (time 1786423589.55 total 124.35 last 11.21) - open/close 1696 (time 1786423599.56 total 134.37 last 17.77) - open/close 1939 (time 1786423609.68 total 144.48 last 24.03) - open/close 2087 (time 1786423620.50 total 155.31 last 13.67) - open/close 2372 (time 1786423630.51 total 165.31 last 28.49) - open/close 2574 (time 1786423640.66 total 175.47 last 19.89) - open/close 2677 (time 1786423650.68 total 185.49 last 10.28) - open/close 2892 (time 1786423660.89 total 195.70 last 21.06) - open/close 3212 (time 1786423671.05 total 205.86 last 31.50) - open/close 3456 (time 1786423683.87 total 218.68 last 19.03) - open/close 3714 (time 1786423694.23 total 229.03 last 24.92) - open/close 3879 (time 1786423705.11 total 239.92 last 15.16) - open/close 4148 (time 1786423715.13 total 249.93 last 26.86) - open/close 4328 (time 1786423725.14 total 259.94 last 17.98) - open/close 4530 (time 1786423735.37 total 270.18 last 19.73) - open/close 4756 (time 1786423745.92 total 280.73 last 21.42) - open/close 4946 (time 1786423755.95 total 290.75 last 18.96) - open/close 5169 (time 1786423766.10 total 300.90 last 21.97) - open/close 5309 (time 1786423776.40 total 311.20 last 13.59) - open/close 5548 (time 1786423786.41 total 321.22 last 23.87) - open/close 5790 (time 1786423796.77 total 331.57 last 23.37) - open/close 6078 (time 1786423807.56 total 342.36 last 26.69) - open/close 6153 (time 1786423817.59 total 352.39 last 7.48) - open/close 6262 (time 1786423828.00 total 362.80 last 10.47) - open/close 6535 (time 1786423838.01 total 372.82 last 27.25) - open/close 6743 (time 1786423848.79 total 383.60 last 19.30) - open/close 6849 (time 1786423860.12 total 394.92 last 9.36) - open/close 7022 (time 1786423870.62 total 405.42 last 16.47) - open/close 7309 (time 1786423880.62 total 415.43 last 28.68) - open/close 7530 (time 1786423890.64 total 425.45 last 22.06) - open/close 7752 (time 1786423901.52 total 436.33 last 20.41) - open/close 7985 (time 1786423911.74 total 446.54 last 22.81) - open/close 8267 (time 1786423921.74 total 456.54 last 28.19) - open/close 8451 (time 1786423932.05 total 466.85 last 17.85) - open/close 8683 (time 1786423942.74 total 477.55 last 21.70) - open/close 8871 (time 1786423952.77 total 487.58 last 18.74) - open/close 9073 (time 1786423963.55 total 498.35 last 18.75) - open/close 9273 (time 1786423974.54 total 509.35 last 18.19) - open/close 9429 (time 1786423984.93 total 519.74 last 15.02) - open/close 9592 (time 1786423997.54 total 532.35 last 12.93) - open/close 9767 (time 1786424008.86 total 543.67 last 15.46) - open/close 10000 (time 1786424019.10 total 553.90 last 22.76) - open/close 10225 (time 1786424029.53 total 564.34 last 21.56) - open/close 10392 (time 1786424040.18 total 574.99 last 15.69) - open/close 10571 (time 1786424052.17 total 586.98 last 14.93) - open/close 10704 (time 1786424063.46 total 598.27 last 11.78) - open/close 10897 (time 1786424073.48 total 608.29 last 19.26) - open/close 11022 (time 1786424084.28 total 619.08 last 11.58) - open/close 11193 (time 1786424095.55 total 630.35 last 15.17) - open/close 11460 (time 1786424105.68 total 640.49 last 26.34) - open/close 11741 (time 1786424116.36 total 651.16 last 26.33) - open/close 11949 (time 1786424126.49 total 661.29 last 20.53) - open/close 12143 (time 1786424137.11 total 671.91 last 18.27) - open/close 12397 (time 1786424147.12 total 681.92 last 25.37) - open/close 12661 (time 1786424157.60 total 692.40 last 25.19) - open/close 12876 (time 1786424168.07 total 702.87 last 20.54) - open/close 13195 (time 1786424178.07 total 712.87 last 31.90) - open/close 13341 (time 1786424189.01 total 723.81 last 13.35) - open/close 13495 (time 1786424205.07 total 739.88 last 9.58) - open/close 13693 (time 1786424215.26 total 750.06 last 19.44) - open/close 13882 (time 1786424226.19 total 760.99 last 17.30) - open/close 14142 (time 1786424236.39 total 771.19 last 25.49) - open/close 14407 (time 1786424246.70 total 781.51 last 25.68) - open/close 14663 (time 1786424257.18 total 791.99 last 24.43) - open/close 14823 (time 1786424267.82 total 802.62 last 15.05) - open/close 14898 (time 1786424278.43 total 813.23 last 7.07) - open/close 15020 (time 1786424290.00 total 824.81 last 10.54) - open/close 15288 (time 1786424300.01 total 834.82 last 26.78) - open/close 15488 (time 1786424310.50 total 845.31 last 19.07) - open/close 15804 (time 1786424320.59 total 855.40 last 31.32) - open/close 16094 (time 1786424330.61 total 865.41 last 28.95) - open/close 16279 (time 1786424341.20 total 876.00 last 17.47) - open/close 16545 (time 1786424351.36 total 886.17 last 26.17) - open/close 16699 (time 1786424361.36 total 896.17 last 15.40) - open/close 16890 (time 1786424371.92 total 906.72 last 18.10) - open/close 17113 (time 1786424382.61 total 917.42 last 20.85) - open/close 17189 (time 1786424392.61 total 927.42 last 7.60) - open/close 17251 (time 1786424402.62 total 937.43 last 6.20) - open/close 17483 (time 1786424412.63 total 947.43 last 23.18) - open/close 17603 (time 1786424422.63 total 957.43 last 12.00) - open/close 17851 (time 1786424433.28 total 968.08 last 23.28) - open/close 17937 (time 1786424443.30 total 978.10 last 8.58) - open/close 18034 (time 1786424453.91 total 988.71 last 9.14) - open/close 18113 (time 1786424463.92 total 998.72 last 7.89) - open/close 18231 (time 1786424476.82 total 1011.62 last 9.15) - open/close 18444 (time 1786424487.37 total 1022.18 last 20.18) - open/close 18820 (time 1786424497.38 total 1032.18 last 37.58) - open/close 19671 (time 1786424507.38 total 1042.18 last 85.07) - open/close 20000 (time 1786424510.99 total 1045.79 last 91.14) - open/close 20868 (time 1786424520.99 total 1055.80 last 86.78) - open/close 21643 (time 1786424531.00 total 1065.81 last 77.43) - open/close 22642 (time 1786424541.00 total 1075.81 last 99.88) - open/close 23618 (time 1786424551.01 total 1085.81 last 97.54) - open/close 24611 (time 1786424561.02 total 1095.82 last 99.23) - open/close 25522 (time 1786424571.02 total 1105.83 last 91.06) - open/close 26471 (time 1786424581.02 total 1115.83 last 94.87) - open/close 27309 (time 1786424591.04 total 1125.84 last 83.69) - open/close 28236 (time 1786424601.04 total 1135.85 last 92.64) - open/close 29210 (time 1786424611.05 total 1145.85 last 97.35) - open/close 30000 (time 1786424620.34 total 1155.15 last 84.98) - open/close 30941 (time 1786424630.35 total 1165.16 last 94.04) - open/close 31929 (time 1786424640.35 total 1175.16 last 98.75) - open/close 32654 (time 1786424650.35 total 1185.16 last 72.50) - open/close 33268 (time 1786424660.37 total 1195.17 last 61.33) - open/close 34002 (time 1786424670.37 total 1205.18 last 73.34) - open/close 34855 (time 1786424680.38 total 1215.18 last 85.29) - open/close 35719 (time 1786424690.38 total 1225.19 last 86.33) - open/close 36559 (time 1786424700.39 total 1235.20 last 83.94) - open/close 37452 (time 1786424710.39 total 1245.20 last 89.30) - open/close 38218 (time 1786424720.40 total 1255.20 last 76.55) - open/close 39065 (time 1786424730.42 total 1265.23 last 84.47) - open/close 40000 (time 1786424740.29 total 1275.10 last 94.74) - open/close 40959 (time 1786424750.32 total 1285.12 last 95.68) - open/close 41688 (time 1786424760.33 total 1295.13 last 72.82) - open/close 42475 (time 1786424770.33 total 1305.14 last 78.66) - open/close 43433 (time 1786424780.33 total 1315.14 last 95.77) - open/close 44362 (time 1786424790.35 total 1325.16 last 92.72) - open/close 45178 (time 1786424800.36 total 1335.16 last 81.56) - open/close 46051 (time 1786424810.36 total 1345.16 last 87.30) - open/close 46924 (time 1786424820.36 total 1355.17 last 87.27) - open/close 47741 (time 1786424830.36 total 1365.17 last 81.69) - open/close 48487 (time 1786424840.37 total 1375.18 last 74.52) - open/close 49234 (time 1786424850.38 total 1385.19 last 74.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1389.49 seconds: 35.72 ops/second - unlinked 0 (time 1786424859 ; total 0 ; last 0) - unlinked 10000 (time 1786424980 ; total 121 ; last 121) - unlinked 20000 (time 1786425086 ; total 227 ; last 106) - unlinked 30000 (time 1786425155 ; total 296 ; last 69) - unlinked 40000 (time 1786425224 ; total 365 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 429 seconds: 115.689980 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg111-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg111-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg111-server stop mds service on oleg111-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg111-server unloading modules via unload_modules_local on: 'oleg111-server' oleg111-server: oleg111-server.virtnet: executing unload_modules_local oleg111-server: modules unloaded. pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-client: ssh exited with exit code 2 pdsh@oleg111-client: oleg111-server: ssh exited with exit code 2