== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:59:23 (1776999563) start mds service on oleg243-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing load_modules_local oleg243-server: Loading modules from /home/green/git/lustre-release/lustre oleg243-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg243-server: detected 4 online CPUs by sysfs oleg243-server: MODOPTS_LIBCFS= oleg243-server: Force libcfs to create 2 CPU partitions oleg243-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg243-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg243-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg243-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 191 (time 1776999663.74 total 10.01 last 19.07) - open/close 471 (time 1776999673.75 total 20.02 last 27.97) - open/close 612 (time 1776999683.77 total 30.04 last 14.08) - open/close 831 (time 1776999694.24 total 40.52 last 20.90) - open/close 988 (time 1776999704.29 total 50.56 last 15.63) - open/close 1256 (time 1776999714.29 total 60.57 last 26.78) - open/close 1522 (time 1776999725.08 total 71.36 last 24.65) - open/close 1777 (time 1776999735.51 total 81.79 last 24.46) - open/close 2001 (time 1776999745.62 total 91.89 last 22.17) - open/close 2240 (time 1776999755.74 total 102.01 last 23.60) - open/close 2410 (time 1776999766.10 total 112.37 last 16.41) - open/close 2708 (time 1776999776.11 total 122.38 last 29.77) - open/close 2950 (time 1776999786.33 total 132.60 last 23.68) - open/close 3181 (time 1776999796.37 total 142.65 last 22.99) - open/close 3418 (time 1776999806.39 total 152.66 last 23.68) - open/close 3656 (time 1776999816.93 total 163.20 last 22.58) - open/close 3906 (time 1776999826.97 total 173.24 last 24.89) - open/close 4109 (time 1776999836.98 total 183.25 last 20.28) - open/close 4315 (time 1776999847.34 total 193.62 last 19.88) - open/close 4530 (time 1776999858.09 total 204.36 last 20.01) - open/close 4783 (time 1776999869.08 total 215.36 last 23.01) - open/close 5024 (time 1776999879.34 total 225.61 last 23.50) - open/close 5273 (time 1776999890.03 total 236.30 last 23.30) - open/close 5533 (time 1776999900.03 total 246.30 last 26.00) - open/close 5757 (time 1776999910.05 total 256.32 last 22.35) - open/close 5999 (time 1776999921.04 total 267.32 last 22.01) - open/close 6240 (time 1776999931.05 total 277.32 last 24.08) - open/close 6486 (time 1776999941.40 total 287.68 last 23.77) - open/close 6737 (time 1776999951.41 total 297.68 last 25.08) - open/close 6980 (time 1776999961.41 total 307.68 last 24.30) - open/close 7188 (time 1776999971.43 total 317.70 last 20.76) - open/close 7370 (time 1776999982.49 total 328.76 last 16.46) - open/close 7562 (time 1776999992.49 total 338.77 last 19.19) - open/close 7823 (time 1777000003.16 total 349.44 last 24.46) - open/close 8069 (time 1777000013.18 total 359.45 last 24.56) - open/close 8313 (time 1777000024.21 total 370.49 last 22.11) - open/close 8588 (time 1777000034.55 total 380.83 last 26.60) - open/close 8802 (time 1777000044.82 total 391.09 last 20.84) - open/close 9005 (time 1777000054.83 total 401.11 last 20.27) - open/close 9237 (time 1777000065.81 total 412.09 last 21.13) - open/close 9480 (time 1777000076.70 total 422.97 last 22.32) - open/close 9677 (time 1777000087.19 total 433.46 last 18.78) - open/close 9919 (time 1777000097.37 total 443.65 last 23.77) - open/close 10000 (time 1777000100.54 total 446.81 last 25.57) - open/close 10205 (time 1777000110.62 total 456.89 last 20.34) - open/close 10472 (time 1777000120.62 total 466.89 last 26.70) - open/close 10650 (time 1777000130.62 total 476.89 last 17.80) - open/close 10857 (time 1777000140.82 total 487.10 last 20.29) - open/close 11112 (time 1777000150.84 total 497.11 last 25.45) - open/close 11306 (time 1777000160.85 total 507.12 last 19.38) - open/close 11484 (time 1777000170.86 total 517.13 last 17.79) - open/close 11615 (time 1777000180.89 total 527.16 last 13.06) - open/close 11804 (time 1777000190.90 total 537.17 last 18.88) - open/close 11937 (time 1777000201.13 total 547.40 last 13.00) - open/close 12138 (time 1777000211.16 total 557.43 last 20.04) - open/close 12351 (time 1777000222.12 total 568.39 last 19.43) - open/close 12561 (time 1777000232.28 total 578.56 last 20.66) - open/close 12764 (time 1777000242.29 total 588.56 last 20.29) - open/close 12981 (time 1777000252.29 total 598.56 last 21.70) - open/close 13120 (time 1777000262.44 total 608.71 last 13.70) - open/close 13282 (time 1777000272.60 total 618.88 last 15.94) - open/close 13450 (time 1777000282.96 total 629.24 last 16.22) - open/close 13570 (time 1777000293.03 total 639.30 last 11.92) - open/close 13763 (time 1777000303.83 total 650.10 last 17.87) - open/close 13940 (time 1777000314.58 total 660.85 last 16.47) - open/close 14121 (time 1777000325.20 total 671.48 last 17.04) - open/close 14390 (time 1777000335.96 total 682.23 last 25.01) - open/close 14567 (time 1777000346.51 total 692.79 last 16.77) - open/close 14693 (time 1777000362.27 total 708.54 last 8.00) - open/close 14870 (time 1777000372.58 total 718.85 last 17.17) - open/close 15017 (time 1777000382.93 total 729.21 last 14.20) - open/close 15171 (time 1777000393.56 total 739.84 last 14.49) - open/close 15368 (time 1777000403.73 total 750.01 last 19.37) - open/close 16357 (time 1777000413.74 total 760.02 last 98.78) - open/close 17427 (time 1777000423.76 total 770.03 last 106.87) - open/close 18562 (time 1777000433.76 total 780.03 last 113.46) - open/close 19689 (time 1777000443.76 total 790.03 last 112.70) - open/close 20000 (time 1777000446.50 total 792.78 last 113.45) - open/close 21111 (time 1777000456.51 total 802.78 last 111.06) - open/close 22233 (time 1777000466.51 total 812.78 last 112.20) - open/close 23242 (time 1777000476.52 total 822.80 last 100.75) - open/close 24193 (time 1777000486.52 total 832.80 last 95.08) - open/close 25122 (time 1777000496.53 total 842.80 last 92.85) - open/close 26212 (time 1777000506.53 total 852.81 last 108.96) - open/close 27348 (time 1777000516.53 total 862.81 last 113.58) - open/close 28437 (time 1777000526.54 total 872.81 last 108.87) - open/close 29402 (time 1777000536.54 total 882.81 last 96.50) - open/close 30000 (time 1777000543.01 total 889.29 last 92.33) - open/close 31076 (time 1777000553.03 total 899.30 last 107.47) - open/close 32093 (time 1777000563.03 total 909.31 last 101.65) - open/close 33118 (time 1777000573.04 total 919.31 last 102.42) - open/close 34202 (time 1777000583.04 total 929.31 last 108.40) - open/close 35254 (time 1777000593.04 total 939.32 last 105.15) - open/close 36215 (time 1777000603.06 total 949.33 last 95.98) - open/close 37170 (time 1777000613.06 total 959.33 last 95.46) - open/close 38117 (time 1777000623.06 total 969.33 last 94.70) - open/close 39123 (time 1777000633.06 total 979.33 last 100.59) - open/close 40000 (time 1777000641.79 total 988.06 last 100.47) - open/close 40867 (time 1777000651.80 total 998.07 last 86.64) - open/close 41869 (time 1777000661.80 total 1008.07 last 100.17) - open/close 42817 (time 1777000671.82 total 1018.09 last 94.65) - open/close 43813 (time 1777000681.82 total 1028.09 last 99.58) - open/close 44740 (time 1777000691.82 total 1038.09 last 92.70) - open/close 45650 (time 1777000701.83 total 1048.10 last 90.94) - open/close 46696 (time 1777000711.83 total 1058.10 last 104.56) - open/close 47687 (time 1777000721.84 total 1068.11 last 98.99) - open/close 48720 (time 1777000731.86 total 1078.13 last 103.10) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1088.10 seconds: 45.61 ops/second - unlinked 0 (time 1777000744 ; total 0 ; last 0) - unlinked 10000 (time 1777000838 ; total 94 ; last 94) - unlinked 20000 (time 1777000901 ; total 157 ; last 63) - unlinked 30000 (time 1777000957 ; total 213 ; last 56) - unlinked 40000 (time 1777001011 ; total 267 ; last 54) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 372 seconds: 133.419357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2