== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:21:50 (1786011710) start mds service on oleg204-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg204-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 128 (time 1786011823.98 total 10.02 last 12.77) - open/close 243 (time 1786011834.67 total 20.71 last 10.75) - open/close 377 (time 1786011844.67 total 30.72 last 13.40) - open/close 594 (time 1786011856.82 total 42.86 last 17.86) - open/close 788 (time 1786011866.83 total 52.87 last 19.39) - open/close 1009 (time 1786011879.66 total 65.70 last 17.22) - open/close 1139 (time 1786011890.24 total 76.28 last 12.29) - open/close 1297 (time 1786011900.33 total 86.37 last 15.66) - open/close 1491 (time 1786011910.33 total 96.38 last 19.39) - open/close 1677 (time 1786011920.34 total 106.38 last 18.60) - open/close 1934 (time 1786011931.38 total 117.42 last 23.28) - open/close 2275 (time 1786011941.39 total 127.44 last 34.04) - open/close 2453 (time 1786011952.25 total 138.29 last 16.40) - open/close 2568 (time 1786011962.27 total 148.31 last 11.47) - open/close 2778 (time 1786011972.29 total 158.33 last 20.97) - open/close 2858 (time 1786011982.84 total 168.88 last 7.58) - open/close 2968 (time 1786011993.09 total 179.13 last 10.74) - open/close 3160 (time 1786012003.50 total 189.54 last 18.43) - open/close 3423 (time 1786012014.07 total 200.11 last 24.89) - open/close 3549 (time 1786012024.45 total 210.49 last 12.14) - open/close 3652 (time 1786012035.26 total 221.30 last 9.53) - open/close 3803 (time 1786012045.68 total 231.72 last 14.49) - open/close 3932 (time 1786012055.68 total 241.73 last 12.89) - open/close 4012 (time 1786012065.96 total 252.00 last 7.78) - open/close 4174 (time 1786012076.91 total 262.95 last 14.80) - open/close 4392 (time 1786012088.44 total 274.48 last 18.91) - open/close 4656 (time 1786012098.45 total 284.49 last 26.39) - open/close 4827 (time 1786012109.26 total 295.30 last 15.81) - open/close 5087 (time 1786012119.27 total 305.31 last 25.98) - open/close 5319 (time 1786012129.81 total 315.86 last 22.00) - open/close 5535 (time 1786012140.76 total 326.80 last 19.74) - open/close 5750 (time 1786012151.47 total 337.51 last 20.07) - open/close 5874 (time 1786012161.47 total 347.52 last 12.39) - open/close 5970 (time 1786012172.22 total 358.26 last 8.93) - open/close 6100 (time 1786012185.16 total 371.20 last 10.05) - open/close 6169 (time 1786012196.08 total 382.12 last 6.32) - open/close 6350 (time 1786012206.10 total 392.14 last 18.06) - open/close 6607 (time 1786012216.15 total 402.19 last 25.57) - open/close 6730 (time 1786012226.42 total 412.46 last 11.98) - open/close 6897 (time 1786012237.32 total 423.36 last 15.33) - open/close 7079 (time 1786012247.32 total 433.36 last 18.19) - open/close 7194 (time 1786012257.32 total 443.36 last 11.50) - open/close 7338 (time 1786012268.18 total 454.22 last 13.27) - open/close 7483 (time 1786012278.58 total 464.62 last 13.94) - open/close 7623 (time 1786012289.54 total 475.58 last 12.77) - open/close 7729 (time 1786012300.48 total 486.52 last 9.69) - open/close 7910 (time 1786012310.50 total 496.54 last 18.07) - open/close 8084 (time 1786012320.84 total 506.88 last 16.82) - open/close 8224 (time 1786012331.52 total 517.56 last 13.12) - open/close 8396 (time 1786012341.53 total 527.57 last 17.17) - open/close 8590 (time 1786012351.54 total 537.58 last 19.39) - open/close 8767 (time 1786012361.55 total 547.60 last 17.67) - open/close 9026 (time 1786012371.57 total 557.61 last 25.87) - open/close 9287 (time 1786012382.29 total 568.33 last 24.35) - open/close 9547 (time 1786012392.29 total 578.33 last 25.99) - open/close 9799 (time 1786012402.47 total 588.51 last 24.75) - open/close 10000 (time 1786012410.21 total 596.25 last 25.97) - open/close 10213 (time 1786012421.17 total 607.22 last 19.43) - open/close 10329 (time 1786012432.12 total 618.16 last 10.60) - open/close 10397 (time 1786012442.37 total 628.42 last 6.63) - open/close 10594 (time 1786012452.98 total 639.02 last 18.57) - open/close 10786 (time 1786012468.11 total 654.15 last 12.69) - open/close 10969 (time 1786012478.92 total 664.96 last 16.93) - open/close 11104 (time 1786012489.24 total 675.28 last 13.08) - open/close 11292 (time 1786012500.15 total 686.19 last 17.23) - open/close 11411 (time 1786012511.13 total 697.17 last 10.84) - open/close 11507 (time 1786012523.06 total 709.10 last 8.04) - open/close 11707 (time 1786012533.09 total 719.13 last 19.95) - open/close 11866 (time 1786012543.11 total 729.15 last 15.87) - open/close 11978 (time 1786012553.22 total 739.26 last 11.08) - open/close 12134 (time 1786012563.70 total 749.74 last 14.88) - open/close 12347 (time 1786012576.36 total 762.40 last 16.83) - open/close 12488 (time 1786012587.79 total 773.83 last 12.34) - open/close 12626 (time 1786012597.88 total 783.93 last 13.67) - open/close 12852 (time 1786012607.91 total 793.95 last 22.55) - open/close 12950 (time 1786012618.87 total 804.91 last 8.94) - open/close 13165 (time 1786012629.89 total 815.93 last 19.50) - open/close 13338 (time 1786012640.32 total 826.36 last 16.60) - open/close 13526 (time 1786012650.33 total 836.37 last 18.77) - open/close 13642 (time 1786012660.46 total 846.50 last 11.46) - open/close 13782 (time 1786012670.46 total 856.50 last 13.99) - open/close 13982 (time 1786012680.47 total 866.51 last 19.99) - open/close 14186 (time 1786012690.92 total 876.96 last 19.52) - open/close 14436 (time 1786012701.94 total 887.98 last 22.68) - open/close 14628 (time 1786012712.12 total 898.16 last 18.87) - open/close 14740 (time 1786012722.39 total 908.43 last 10.90) - open/close 14879 (time 1786012732.39 total 918.43 last 13.90) - open/close 15056 (time 1786012742.85 total 928.89 last 16.94) - open/close 15140 (time 1786012752.85 total 938.89 last 8.40) - open/close 15378 (time 1786012763.52 total 949.56 last 22.30) - open/close 15632 (time 1786012773.53 total 959.58 last 25.36) - open/close 15853 (time 1786012783.94 total 969.98 last 21.24) - open/close 16126 (time 1786012793.95 total 979.99 last 27.27) - open/close 16339 (time 1786012804.14 total 990.18 last 20.90) - open/close 16571 (time 1786012814.15 total 1000.19 last 23.18) - open/close 16756 (time 1786012824.17 total 1010.22 last 18.45) - open/close 17033 (time 1786012837.03 total 1023.07 last 21.54) - open/close 17157 (time 1786012847.05 total 1033.09 last 12.38) - open/close 17377 (time 1786012857.06 total 1043.10 last 21.97) - open/close 17518 (time 1786012867.56 total 1053.60 last 13.44) - open/close 17731 (time 1786012877.56 total 1063.60 last 21.30) - open/close 18234 (time 1786012887.57 total 1073.61 last 50.25) - open/close 19035 (time 1786012897.58 total 1083.62 last 80.02) - open/close 19829 (time 1786012907.58 total 1093.62 last 79.36) - open/close 20657 (time 1786012917.58 total 1103.63 last 82.77) - open/close 21399 (time 1786012927.60 total 1113.65 last 74.05) - open/close 22098 (time 1786012937.61 total 1123.65 last 69.88) - open/close 22892 (time 1786012947.63 total 1133.67 last 79.24) - open/close 23556 (time 1786012957.65 total 1143.69 last 66.27) - open/close 24251 (time 1786012967.65 total 1153.69 last 69.49) - open/close 24987 (time 1786012977.65 total 1163.69 last 73.59) - open/close 25873 (time 1786012987.65 total 1173.70 last 88.56) - open/close 26649 (time 1786012997.67 total 1183.71 last 77.49) - open/close 27502 (time 1786013007.68 total 1193.72 last 85.21) - open/close 28228 (time 1786013017.68 total 1203.73 last 72.57) - open/close 29043 (time 1786013027.69 total 1213.73 last 81.42) - open/close 29936 (time 1786013037.70 total 1223.74 last 89.26) - open/close 30856 (time 1786013047.70 total 1233.74 last 91.97) - open/close 31791 (time 1786013057.72 total 1243.76 last 93.30) - open/close 32650 (time 1786013067.73 total 1253.78 last 85.79) - open/close 33521 (time 1786013077.77 total 1263.81 last 86.79) - open/close 34393 (time 1786013087.78 total 1273.82 last 87.09) - open/close 35170 (time 1786013097.79 total 1283.83 last 77.67) - open/close 35932 (time 1786013107.79 total 1293.83 last 76.17) - open/close 36738 (time 1786013117.80 total 1303.84 last 80.56) - open/close 37420 (time 1786013127.80 total 1313.84 last 68.19) - open/close 38122 (time 1786013137.80 total 1323.85 last 70.15) - open/close 38913 (time 1786013147.82 total 1333.86 last 79.02) - open/close 39500 (time 1786013157.83 total 1343.88 last 58.59) - open/close 40000 (time 1786013165.91 total 1351.95 last 61.91) - open/close 40841 (time 1786013175.92 total 1361.96 last 84.03) - open/close 41741 (time 1786013185.92 total 1371.96 last 89.99) - open/close 42544 (time 1786013195.93 total 1381.97 last 80.26) - open/close 43339 (time 1786013205.94 total 1391.98 last 79.41) - open/close 43942 (time 1786013215.94 total 1401.98 last 60.26) - open/close 44598 (time 1786013225.94 total 1411.98 last 65.60) - open/close 45247 (time 1786013235.95 total 1421.99 last 64.84) - open/close 45877 (time 1786013245.96 total 1432.00 last 62.96) - open/close 46470 (time 1786013255.97 total 1442.01 last 59.26) - open/close 47169 (time 1786013265.97 total 1452.01 last 69.85) - open/close 47991 (time 1786013275.98 total 1462.02 last 82.15) - open/close 48779 (time 1786013285.99 total 1472.03 last 78.73) - open/close 49583 (time 1786013296.00 total 1482.04 last 80.31) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1482.72 seconds: 33.47 ops/second - unlinked 0 (time 1786013301 ; total 0 ; last 0) - unlinked 10000 (time 1786013446 ; total 145 ; last 145) - unlinked 20000 (time 1786013599 ; total 298 ; last 153) - unlinked 30000 (time 1786013762 ; total 461 ; last 163) - unlinked 40000 (time 1786013910 ; total 609 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 743 seconds: 66.798119 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: modules unloaded. pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2