== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:10:53 (1787847053) start mds service on oleg312-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing load_modules_local oleg312-server: Loading modules from /home/green/git/lustre-release/lustre oleg312-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg312-server: detected 4 online CPUs by sysfs oleg312-server: MODOPTS_LIBCFS= oleg312-server: Force libcfs to create 2 CPU partitions oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg312-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg312-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg312-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 276 (time 1787847161.19 total 10.65 last 25.92) - open/close 479 (time 1787847171.21 total 20.67 last 20.25) - open/close 710 (time 1787847181.38 total 30.84 last 22.72) - open/close 887 (time 1787847191.70 total 41.16 last 17.15) - open/close 986 (time 1787847201.99 total 51.46 last 9.62) - open/close 1227 (time 1787847212.00 total 61.46 last 24.09) - open/close 1468 (time 1787847222.45 total 71.91 last 23.05) - open/close 1728 (time 1787847232.46 total 81.92 last 25.98) - open/close 2004 (time 1787847242.46 total 91.93 last 27.59) - open/close 2236 (time 1787847252.82 total 102.29 last 22.39) - open/close 2396 (time 1787847262.84 total 112.31 last 15.97) - open/close 2667 (time 1787847272.86 total 122.33 last 27.05) - open/close 2872 (time 1787847282.95 total 132.41 last 20.32) - open/close 2988 (time 1787847295.27 total 144.73 last 9.42) - open/close 3129 (time 1787847305.28 total 154.74 last 14.08) - open/close 3275 (time 1787847315.53 total 164.99 last 14.24) - open/close 3445 (time 1787847325.56 total 175.02 last 16.96) - open/close 3651 (time 1787847336.29 total 185.76 last 19.18) - open/close 3793 (time 1787847347.15 total 196.61 last 13.08) - open/close 3938 (time 1787847357.15 total 206.61 last 14.50) - open/close 4119 (time 1787847367.17 total 216.64 last 18.06) - open/close 4267 (time 1787847377.36 total 226.82 last 14.54) - open/close 4380 (time 1787847387.86 total 237.32 last 10.76) - open/close 4598 (time 1787847397.89 total 247.35 last 21.74) - open/close 4774 (time 1787847410.49 total 259.95 last 13.97) - open/close 4889 (time 1787847421.51 total 270.97 last 10.43) - open/close 5021 (time 1787847432.21 total 281.67 last 12.34) - open/close 5092 (time 1787847442.29 total 291.75 last 7.04) - open/close 5272 (time 1787847452.30 total 301.76 last 17.98) - open/close 5403 (time 1787847462.33 total 311.79 last 13.06) - open/close 5551 (time 1787847472.56 total 322.02 last 14.46) - open/close 5853 (time 1787847482.76 total 332.22 last 29.60) - open/close 6126 (time 1787847492.76 total 342.22 last 27.29) - open/close 6332 (time 1787847503.62 total 353.09 last 18.97) - open/close 6527 (time 1787847513.64 total 363.10 last 19.47) - open/close 6708 (time 1787847523.93 total 373.39 last 17.60) - open/close 6849 (time 1787847533.95 total 383.41 last 14.07) - open/close 7130 (time 1787847543.95 total 393.41 last 28.10) - open/close 7372 (time 1787847554.06 total 403.52 last 23.92) - open/close 7572 (time 1787847565.06 total 414.52 last 18.19) - open/close 7795 (time 1787847575.55 total 425.02 last 21.25) - open/close 7956 (time 1787847586.12 total 435.58 last 15.24) - open/close 8134 (time 1787847596.63 total 446.10 last 16.93) - open/close 8340 (time 1787847606.64 total 456.10 last 20.59) - open/close 8583 (time 1787847616.98 total 466.45 last 23.48) - open/close 8818 (time 1787847627.00 total 476.46 last 23.46) - open/close 9088 (time 1787847639.83 total 489.30 last 21.04) - open/close 9388 (time 1787847650.70 total 500.16 last 27.61) - open/close 9636 (time 1787847661.00 total 510.47 last 24.07) - open/close 9793 (time 1787847671.02 total 520.48 last 15.67) - open/close 9952 (time 1787847681.03 total 530.49 last 15.89) - open/close 10000 (time 1787847683.04 total 532.50 last 23.91) - open/close 10236 (time 1787847693.04 total 542.50 last 23.59) - open/close 10420 (time 1787847704.09 total 553.56 last 16.65) - open/close 10678 (time 1787847714.43 total 563.90 last 24.95) - open/close 10960 (time 1787847724.82 total 574.28 last 27.16) - open/close 11246 (time 1787847734.83 total 584.29 last 28.56) - open/close 11380 (time 1787847744.83 total 594.29 last 13.40) - open/close 11576 (time 1787847755.21 total 604.67 last 18.89) - open/close 11723 (time 1787847768.04 total 617.50 last 11.46) - open/close 11822 (time 1787847778.05 total 627.51 last 9.89) - open/close 11968 (time 1787847788.26 total 637.72 last 14.29) - open/close 12149 (time 1787847798.35 total 647.81 last 17.94) - open/close 12248 (time 1787847808.38 total 657.84 last 9.87) - open/close 12378 (time 1787847818.45 total 667.91 last 12.90) - open/close 12537 (time 1787847828.61 total 678.07 last 15.66) - open/close 12769 (time 1787847839.19 total 688.66 last 21.92) - open/close 13089 (time 1787847850.16 total 699.62 last 29.17) - open/close 13359 (time 1787847860.62 total 710.08 last 25.82) - open/close 13566 (time 1787847871.05 total 720.51 last 19.85) - open/close 13874 (time 1787847881.05 total 730.51 last 30.80) - open/close 14178 (time 1787847891.48 total 740.94 last 29.15) - open/close 14432 (time 1787847902.12 total 751.58 last 23.86) - open/close 14546 (time 1787847912.59 total 762.05 last 10.89) - open/close 14758 (time 1787847923.22 total 772.69 last 19.94) - open/close 15026 (time 1787847933.71 total 783.18 last 25.55) - open/close 15268 (time 1787847944.46 total 793.92 last 22.53) - open/close 15499 (time 1787847954.51 total 803.98 last 22.97) - open/close 15745 (time 1787847965.01 total 814.47 last 23.45) - open/close 15949 (time 1787847977.07 total 826.53 last 16.91) - open/close 16193 (time 1787847987.99 total 837.46 last 22.34) - open/close 16446 (time 1787847998.57 total 848.03 last 23.92) - open/close 16600 (time 1787848008.57 total 858.04 last 15.40) - open/close 16849 (time 1787848018.58 total 868.04 last 24.89) - open/close 17015 (time 1787848028.62 total 878.08 last 16.52) - open/close 17267 (time 1787848038.63 total 888.10 last 25.17) - open/close 17509 (time 1787848048.64 total 898.10 last 24.19) - open/close 17682 (time 1787848058.64 total 908.10 last 17.29) - open/close 17769 (time 1787848069.59 total 919.06 last 7.94) - open/close 18033 (time 1787848079.61 total 929.07 last 26.36) - open/close 18247 (time 1787848090.22 total 939.68 last 20.16) - open/close 18503 (time 1787848100.74 total 950.21 last 24.32) - open/close 18761 (time 1787848111.01 total 960.48 last 25.13) - open/close 19056 (time 1787848121.93 total 971.40 last 27.01) - open/close 19376 (time 1787848132.76 total 982.22 last 29.56) - open/close 20000 (time 1787848141.27 total 990.73 last 73.33) - open/close 20907 (time 1787848151.28 total 1000.74 last 90.60) - open/close 21749 (time 1787848161.28 total 1010.75 last 84.19) - open/close 22602 (time 1787848171.28 total 1020.75 last 85.29) - open/close 23466 (time 1787848181.29 total 1030.75 last 86.34) - open/close 24298 (time 1787848191.30 total 1040.76 last 83.15) - open/close 25065 (time 1787848201.31 total 1050.77 last 76.63) - open/close 25969 (time 1787848211.31 total 1060.77 last 90.37) - open/close 26817 (time 1787848221.31 total 1070.78 last 84.78) - open/close 27684 (time 1787848231.32 total 1080.78 last 86.63) - open/close 28580 (time 1787848241.33 total 1090.79 last 89.54) - open/close 29426 (time 1787848251.33 total 1100.79 last 84.57) - open/close 30000 (time 1787848259.13 total 1108.59 last 73.64) - open/close 30691 (time 1787848269.13 total 1118.60 last 69.05) - open/close 31441 (time 1787848279.14 total 1128.60 last 74.94) - open/close 32194 (time 1787848289.14 total 1138.61 last 75.27) - open/close 32946 (time 1787848299.15 total 1148.61 last 75.16) - open/close 33615 (time 1787848309.15 total 1158.62 last 66.86) - open/close 34318 (time 1787848319.16 total 1168.63 last 70.24) - open/close 35121 (time 1787848329.17 total 1178.63 last 80.23) - open/close 35971 (time 1787848339.18 total 1188.64 last 84.96) - open/close 36901 (time 1787848349.19 total 1198.65 last 92.91) - open/close 37760 (time 1787848359.20 total 1208.66 last 85.79) - open/close 38622 (time 1787848369.20 total 1218.67 last 86.15) - open/close 39523 (time 1787848379.22 total 1228.68 last 90.00) - open/close 40000 (time 1787848385.18 total 1234.65 last 79.94) - open/close 40782 (time 1787848395.19 total 1244.65 last 78.14) - open/close 41472 (time 1787848405.21 total 1254.67 last 68.85) - open/close 42329 (time 1787848415.23 total 1264.69 last 85.58) - open/close 43275 (time 1787848425.23 total 1274.69 last 94.59) - open/close 44111 (time 1787848435.23 total 1284.70 last 83.55) - open/close 44800 (time 1787848445.24 total 1294.70 last 68.87) - open/close 45643 (time 1787848455.25 total 1304.71 last 84.21) - open/close 46492 (time 1787848465.26 total 1314.72 last 84.82) - open/close 47473 (time 1787848475.27 total 1324.73 last 97.96) - open/close 48245 (time 1787848485.28 total 1334.74 last 77.14) - open/close 49102 (time 1787848495.28 total 1344.74 last 85.67) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1351.28 seconds: 36.73 ops/second - unlinked 0 (time 1787848505 ; total 0 ; last 0) - unlinked 10000 (time 1787848621 ; total 116 ; last 116) - unlinked 20000 (time 1787848736 ; total 231 ; last 115) - unlinked 30000 (time 1787848803 ; total 298 ; last 67) - unlinked 40000 (time 1787848872 ; total 367 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 426 seconds: 116.504692 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg312-server unloading modules via unload_modules_local on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local oleg312-server: modules unloaded. pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2