== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:27:19 (1787365639) start mds service on oleg429-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing load_modules_local oleg429-server: Loading modules from /home/green/git/lustre-release/lustre oleg429-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg429-server: detected 4 online CPUs by sysfs oleg429-server: MODOPTS_LIBCFS= oleg429-server: Force libcfs to create 2 CPU partitions oleg429-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg429-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg429-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg429-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock 192.168.204.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 232 (time 1787365756.07 total 10.48 last 22.13) - open/close 476 (time 1787365766.44 total 20.85 last 23.53) - open/close 701 (time 1787365776.61 total 31.02 last 22.12) - open/close 858 (time 1787365787.10 total 41.52 last 14.96) - open/close 1064 (time 1787365799.27 total 53.69 last 16.93) - open/close 1225 (time 1787365811.96 total 66.38 last 12.69) - open/close 1330 (time 1787365827.24 total 81.66 last 6.87) - open/close 1462 (time 1787365837.25 total 91.66 last 13.19) - open/close 1667 (time 1787365847.26 total 101.67 last 20.48) - open/close 1865 (time 1787365857.26 total 111.68 last 19.79) - open/close 2019 (time 1787365867.55 total 121.96 last 14.98) - open/close 2296 (time 1787365877.61 total 132.02 last 27.53) - open/close 2530 (time 1787365887.91 total 142.32 last 22.72) - open/close 2659 (time 1787365900.19 total 154.61 last 10.50) - open/close 2752 (time 1787365910.59 total 165.00 last 8.95) - open/close 2853 (time 1787365921.32 total 175.73 last 9.41) - open/close 3072 (time 1787365931.55 total 185.96 last 21.41) - open/close 3265 (time 1787365942.56 total 196.97 last 17.53) - open/close 3419 (time 1787365954.21 total 208.63 last 13.21) - open/close 3603 (time 1787365965.07 total 219.49 last 16.95) - open/close 3804 (time 1787365975.27 total 229.69 last 19.70) - open/close 4021 (time 1787365986.14 total 240.55 last 19.97) - open/close 4313 (time 1787365997.16 total 251.57 last 26.50) - open/close 4522 (time 1787366007.17 total 261.59 last 20.87) - open/close 4734 (time 1787366017.18 total 271.60 last 21.18) - open/close 4878 (time 1787366028.26 total 282.67 last 13.00) - open/close 5091 (time 1787366038.43 total 292.85 last 20.93) - open/close 5265 (time 1787366048.44 total 302.85 last 17.40) - open/close 5387 (time 1787366059.11 total 313.53 last 11.43) - open/close 5653 (time 1787366069.12 total 323.53 last 26.59) - open/close 5915 (time 1787366079.12 total 333.54 last 26.18) - open/close 6070 (time 1787366089.13 total 343.55 last 15.48) - open/close 6297 (time 1787366099.29 total 353.70 last 22.35) - open/close 6553 (time 1787366110.19 total 364.60 last 23.49) - open/close 6800 (time 1787366120.87 total 375.29 last 23.11) - open/close 7048 (time 1787366130.89 total 385.31 last 24.75) - open/close 7310 (time 1787366141.79 total 396.21 last 24.04) - open/close 7525 (time 1787366151.81 total 406.23 last 21.45) - open/close 7657 (time 1787366162.40 total 416.81 last 12.47) - open/close 7914 (time 1787366173.28 total 427.69 last 23.62) - open/close 8157 (time 1787366185.01 total 439.42 last 20.72) - open/close 8364 (time 1787366195.88 total 450.30 last 19.04) - open/close 8647 (time 1787366205.89 total 460.31 last 28.26) - open/close 8850 (time 1787366215.90 total 470.32 last 20.29) - open/close 9142 (time 1787366225.90 total 480.32 last 29.20) - open/close 9392 (time 1787366236.32 total 490.74 last 23.99) - open/close 9596 (time 1787366246.33 total 500.74 last 20.39) - open/close 9890 (time 1787366256.56 total 510.98 last 28.73) - open/close 10000 (time 1787366260.62 total 515.03 last 27.13) - open/close 10229 (time 1787366270.88 total 525.29 last 22.32) - open/close 10446 (time 1787366280.89 total 535.31 last 21.67) - open/close 10629 (time 1787366293.53 total 547.94 last 14.48) - open/close 10871 (time 1787366304.23 total 558.64 last 22.61) - open/close 11019 (time 1787366314.30 total 568.72 last 14.69) - open/close 11165 (time 1787366325.23 total 579.65 last 13.36) - open/close 11414 (time 1787366335.24 total 589.65 last 24.88) - open/close 11564 (time 1787366345.25 total 599.66 last 14.99) - open/close 11823 (time 1787366356.14 total 610.56 last 23.77) - open/close 12000 (time 1787366366.83 total 621.24 last 16.56) - open/close 12152 (time 1787366377.19 total 631.61 last 14.67) - open/close 12353 (time 1787366387.22 total 641.63 last 20.05) - open/close 12550 (time 1787366397.22 total 651.64 last 19.69) - open/close 12790 (time 1787366407.23 total 661.64 last 23.99) - open/close 12995 (time 1787366417.23 total 671.65 last 20.49) - open/close 13205 (time 1787366428.12 total 682.54 last 19.28) - open/close 13471 (time 1787366438.13 total 692.54 last 26.60) - open/close 13716 (time 1787366448.48 total 702.89 last 23.67) - open/close 14007 (time 1787366459.23 total 713.65 last 27.05) - open/close 14235 (time 1787366470.17 total 724.58 last 20.85) - open/close 14451 (time 1787366480.17 total 734.59 last 21.60) - open/close 14718 (time 1787366490.61 total 745.02 last 25.58) - open/close 14986 (time 1787366501.30 total 755.71 last 25.08) - open/close 15212 (time 1787366511.60 total 766.01 last 21.94) - open/close 15468 (time 1787366521.61 total 776.03 last 25.56) - open/close 15621 (time 1787366532.21 total 786.62 last 14.44) - open/close 15763 (time 1787366542.22 total 796.63 last 14.19) - open/close 15996 (time 1787366552.49 total 806.90 last 22.69) - open/close 16175 (time 1787366563.12 total 817.53 last 16.84) - open/close 16335 (time 1787366573.36 total 827.78 last 15.62) - open/close 16644 (time 1787366584.17 total 838.58 last 28.59) - open/close 16903 (time 1787366594.17 total 848.59 last 25.89) - open/close 17171 (time 1787366604.97 total 859.38 last 24.83) - open/close 17372 (time 1787366614.97 total 869.38 last 20.10) - open/close 17560 (time 1787366624.99 total 879.40 last 18.77) - open/close 17716 (time 1787366637.38 total 891.79 last 12.59) - open/close 17855 (time 1787366647.47 total 901.88 last 13.78) - open/close 18087 (time 1787366658.84 total 913.26 last 20.39) - open/close 18195 (time 1787366670.31 total 924.72 last 9.42) - open/close 18436 (time 1787366680.32 total 934.73 last 24.07) - open/close 18644 (time 1787366690.80 total 945.22 last 19.84) - open/close 18785 (time 1787366700.81 total 955.22 last 14.09) - open/close 18984 (time 1787366710.83 total 965.25 last 19.85) - open/close 19430 (time 1787366720.85 total 975.26 last 44.52) - open/close 20000 (time 1787366728.71 total 983.12 last 72.52) - open/close 20595 (time 1787366738.71 total 993.13 last 59.48) - open/close 21292 (time 1787366748.72 total 1003.14 last 69.64) - open/close 22023 (time 1787366758.73 total 1013.15 last 73.01) - open/close 22768 (time 1787366768.74 total 1023.15 last 74.47) - open/close 23517 (time 1787366778.74 total 1033.15 last 74.89) - open/close 24241 (time 1787366788.74 total 1043.16 last 72.39) - open/close 24967 (time 1787366798.75 total 1053.16 last 72.56) - open/close 25680 (time 1787366808.76 total 1063.17 last 71.24) - open/close 26403 (time 1787366818.78 total 1073.19 last 72.15) - open/close 27101 (time 1787366828.78 total 1083.19 last 69.77) - open/close 27760 (time 1787366838.80 total 1093.22 last 65.74) - open/close 28460 (time 1787366848.81 total 1103.23 last 69.96) - open/close 29094 (time 1787366858.82 total 1113.23 last 63.34) - open/close 29770 (time 1787366868.82 total 1123.23 last 67.60) - open/close 30000 (time 1787366872.06 total 1126.47 last 71.02) - open/close 30681 (time 1787366882.07 total 1136.49 last 68.00) - open/close 31322 (time 1787366892.08 total 1146.50 last 64.05) - open/close 31989 (time 1787366902.09 total 1156.51 last 66.63) - open/close 32654 (time 1787366912.11 total 1166.52 last 66.39) - open/close 33361 (time 1787366922.11 total 1176.52 last 70.69) - open/close 34048 (time 1787366932.11 total 1186.52 last 68.69) - open/close 34813 (time 1787366942.12 total 1196.53 last 76.45) - open/close 35463 (time 1787366952.12 total 1206.54 last 64.94) - open/close 36123 (time 1787366962.13 total 1216.54 last 65.99) - open/close 36821 (time 1787366972.13 total 1226.55 last 69.76) - open/close 37536 (time 1787366982.14 total 1236.56 last 71.45) - open/close 38215 (time 1787366992.14 total 1246.56 last 67.88) - open/close 38787 (time 1787367002.17 total 1256.58 last 57.06) - open/close 39446 (time 1787367012.17 total 1266.58 last 65.89) - open/close 40000 (time 1787367020.33 total 1274.75 last 67.86) - open/close 40677 (time 1787367030.34 total 1284.75 last 67.68) - open/close 41362 (time 1787367040.35 total 1294.76 last 68.42) - open/close 42065 (time 1787367050.35 total 1304.76 last 70.30) - open/close 42780 (time 1787367060.35 total 1314.76 last 71.49) - open/close 43595 (time 1787367070.37 total 1324.78 last 81.35) - open/close 44372 (time 1787367080.39 total 1334.80 last 77.57) - open/close 45043 (time 1787367090.41 total 1344.82 last 66.97) - open/close 45709 (time 1787367100.41 total 1354.82 last 66.57) - open/close 46350 (time 1787367110.42 total 1364.83 last 64.06) - open/close 47065 (time 1787367120.42 total 1374.83 last 71.48) - open/close 47763 (time 1787367130.42 total 1384.84 last 69.77) - open/close 48402 (time 1787367140.43 total 1394.84 last 63.88) - open/close 49071 (time 1787367150.43 total 1404.84 last 66.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1413.11 seconds: 35.12 ops/second - unlinked 0 (time 1787367163 ; total 0 ; last 0) - unlinked 10000 (time 1787367314 ; total 151 ; last 151) - unlinked 20000 (time 1787367467 ; total 304 ; last 153) - unlinked 30000 (time 1787367616 ; total 453 ; last 149) - unlinked 40000 (time 1787367770 ; total 607 ; last 154) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 744 seconds: 66.708336 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server unloading modules via unload_modules_local on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local oleg429-server: modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2