== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:29:16 (1780385356) start mds service on oleg220-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing load_modules_local oleg220-server: Loading modules from /home/green/git/lustre-release/lustre oleg220-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg220-server: detected 4 online CPUs by sysfs oleg220-server: MODOPTS_LIBCFS= oleg220-server: Force libcfs to create 2 CPU partitions oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg220-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg220-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg220-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1 Started lustre-OST0000 oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg220-client.virtnet: -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 279 (time 1780385464.21 total 10.01 last 27.87) - open/close 558 (time 1780385474.22 total 20.02 last 27.89) - open/close 768 (time 1780385484.98 total 30.79 last 19.50) - open/close 1060 (time 1780385494.99 total 40.79 last 29.19) - open/close 1278 (time 1780385505.07 total 50.88 last 21.62) - open/close 1605 (time 1780385515.08 total 60.88 last 32.67) - open/close 1859 (time 1780385527.78 total 73.58 last 20.01) - open/close 2009 (time 1780385537.79 total 83.59 last 14.98) - open/close 2289 (time 1780385547.80 total 93.60 last 27.97) - open/close 2518 (time 1780385558.52 total 104.32 last 21.37) - open/close 2792 (time 1780385568.52 total 114.32 last 27.40) - open/close 3057 (time 1780385578.52 total 124.32 last 26.49) - open/close 3300 (time 1780385588.53 total 134.33 last 24.29) - open/close 3468 (time 1780385598.56 total 144.36 last 16.74) - open/close 3728 (time 1780385608.74 total 154.54 last 25.53) - open/close 3995 (time 1780385619.01 total 164.81 last 26.01) - open/close 4308 (time 1780385629.56 total 175.36 last 29.67) - open/close 4527 (time 1780385639.56 total 185.36 last 21.90) - open/close 4771 (time 1780385652.67 total 198.47 last 18.61) - open/close 4978 (time 1780385662.69 total 208.49 last 20.67) - open/close 5166 (time 1780385673.20 total 219.00 last 17.88) - open/close 5375 (time 1780385683.75 total 229.55 last 19.80) - open/close 5616 (time 1780385694.32 total 240.12 last 22.80) - open/close 5888 (time 1780385704.88 total 250.69 last 25.75) - open/close 6113 (time 1780385714.90 total 260.70 last 22.47) - open/close 6371 (time 1780385724.92 total 270.72 last 25.76) - open/close 6617 (time 1780385735.08 total 280.88 last 24.19) - open/close 6857 (time 1780385745.09 total 290.89 last 23.98) - open/close 7058 (time 1780385757.18 total 302.98 last 16.63) - open/close 7325 (time 1780385768.05 total 313.85 last 24.57) - open/close 7621 (time 1780385778.92 total 324.72 last 27.23) - open/close 7867 (time 1780385788.92 total 334.72 last 24.60) - open/close 8161 (time 1780385798.96 total 344.76 last 29.28) - open/close 8460 (time 1780385809.53 total 355.33 last 28.29) - open/close 8677 (time 1780385819.81 total 365.61 last 21.11) - open/close 8952 (time 1780385829.83 total 375.63 last 27.45) - open/close 9191 (time 1780385840.24 total 386.04 last 22.95) - open/close 9434 (time 1780385850.86 total 396.66 last 22.89) - open/close 9685 (time 1780385861.83 total 407.63 last 22.87) - open/close 9898 (time 1780385872.05 total 417.85 last 20.85) - open/close 10000 (time 1780385875.54 total 421.34 last 29.22) - open/close 10249 (time 1780385886.51 total 432.31 last 22.71) - open/close 10482 (time 1780385897.51 total 443.31 last 21.18) - open/close 10698 (time 1780385908.33 total 454.13 last 19.97) - open/close 10920 (time 1780385918.33 total 464.13 last 22.19) - open/close 11112 (time 1780385929.25 total 475.05 last 17.58) - open/close 11274 (time 1780385939.51 total 485.31 last 15.79) - open/close 11494 (time 1780385949.51 total 495.31 last 21.99) - open/close 11700 (time 1780385959.52 total 505.33 last 20.57) - open/close 11917 (time 1780385969.63 total 515.43 last 21.47) - open/close 12195 (time 1780385979.63 total 525.43 last 27.80) - open/close 12425 (time 1780385990.31 total 536.11 last 21.54) - open/close 12694 (time 1780386001.06 total 546.86 last 25.02) - open/close 12896 (time 1780386011.07 total 556.87 last 20.18) - open/close 13165 (time 1780386022.06 total 567.86 last 24.48) - open/close 13382 (time 1780386032.23 total 578.03 last 21.34) - open/close 13618 (time 1780386043.23 total 589.03 last 21.45) - open/close 13904 (time 1780386053.67 total 599.47 last 27.42) - open/close 14160 (time 1780386063.68 total 609.48 last 25.55) - open/close 14391 (time 1780386074.23 total 620.03 last 21.90) - open/close 14646 (time 1780386084.48 total 630.28 last 24.87) - open/close 14903 (time 1780386094.49 total 640.29 last 25.68) - open/close 15078 (time 1780386104.51 total 650.31 last 17.47) - open/close 15314 (time 1780386115.31 total 661.11 last 21.84) - open/close 15525 (time 1780386125.94 total 671.74 last 19.85) - open/close 15832 (time 1780386136.94 total 682.74 last 27.92) - open/close 16049 (time 1780386147.40 total 693.20 last 20.75) - open/close 16252 (time 1780386157.61 total 703.41 last 19.87) - open/close 16447 (time 1780386168.62 total 714.42 last 17.71) - open/close 16641 (time 1780386179.62 total 725.42 last 17.64) - open/close 16827 (time 1780386190.12 total 735.92 last 17.71) - open/close 17091 (time 1780386201.83 total 747.63 last 22.55) - open/close 17300 (time 1780386211.96 total 757.76 last 20.63) - open/close 17584 (time 1780386222.76 total 768.56 last 26.30) - open/close 17763 (time 1780386233.38 total 779.18 last 16.85) - open/close 17966 (time 1780386243.38 total 789.18 last 20.29) - open/close 18235 (time 1780386254.39 total 800.19 last 24.43) - open/close 19210 (time 1780386264.41 total 810.22 last 97.29) - open/close 20000 (time 1780386272.01 total 817.81 last 104.00) - open/close 20995 (time 1780386282.01 total 827.81 last 99.49) - open/close 22084 (time 1780386292.02 total 837.82 last 108.82) - open/close 23065 (time 1780386302.02 total 847.82 last 98.06) - open/close 23944 (time 1780386312.04 total 857.84 last 87.79) - open/close 24944 (time 1780386322.05 total 867.85 last 99.90) - open/close 26081 (time 1780386332.05 total 877.85 last 113.67) - open/close 27150 (time 1780386342.05 total 887.85 last 106.88) - open/close 28197 (time 1780386352.05 total 897.85 last 104.68) - open/close 29242 (time 1780386362.06 total 907.86 last 104.43) - open/close 30000 (time 1780386369.12 total 914.92 last 107.30) - open/close 30997 (time 1780386379.13 total 924.93 last 99.65) - open/close 31975 (time 1780386389.13 total 934.93 last 97.77) - open/close 32913 (time 1780386399.13 total 944.93 last 93.78) - open/close 33987 (time 1780386409.13 total 954.93 last 107.38) - open/close 35066 (time 1780386419.14 total 964.94 last 107.86) - open/close 36166 (time 1780386429.14 total 974.94 last 109.98) - open/close 37471 (time 1780386439.14 total 984.94 last 130.50) - open/close 38634 (time 1780386449.14 total 994.94 last 116.28) - open/close 40000 (time 1780386458.59 total 1004.39 last 144.53) - open/close 41420 (time 1780386468.60 total 1014.40 last 141.93) - open/close 42790 (time 1780386478.60 total 1024.40 last 137.00) - open/close 44232 (time 1780386488.60 total 1034.40 last 144.20) - open/close 45445 (time 1780386498.60 total 1044.40 last 121.27) - open/close 46774 (time 1780386508.61 total 1054.41 last 132.78) - open/close 48313 (time 1780386518.62 total 1064.42 last 153.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1074.13 seconds: 46.21 ops/second - unlinked 0 (time 1780386530 ; total 0 ; last 0) - unlinked 10000 (time 1780386600 ; total 70 ; last 70) - unlinked 20000 (time 1780386652 ; total 122 ; last 52) - unlinked 30000 (time 1780386691 ; total 161 ; last 39) - unlinked 40000 (time 1780386734 ; total 204 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 245 seconds: 202.575516 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg220-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg220-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server stop mds service on oleg220-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg220-server unloading modules via unload_modules_local on: 'oleg220-server' oleg220-server: oleg220-server.virtnet: executing unload_modules_local oleg220-server: modules unloaded. pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2 pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2