== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:05:14 (1783951514) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1783951624.65 total 10.45 last 15.89) - open/close 320 (time 1783951637.15 total 22.95 last 12.31) - open/close 525 (time 1783951647.45 total 33.25 last 19.91) - open/close 751 (time 1783951658.20 total 44.00 last 21.01) - open/close 912 (time 1783951671.55 total 57.35 last 12.06) - open/close 1117 (time 1783951681.56 total 67.36 last 20.48) - open/close 1252 (time 1783951691.64 total 77.44 last 13.39) - open/close 1394 (time 1783951702.04 total 87.84 last 13.66) - open/close 1522 (time 1783951712.06 total 97.86 last 12.77) - open/close 1637 (time 1783951722.58 total 108.38 last 10.93) - open/close 1800 (time 1783951733.20 total 119.00 last 15.36) - open/close 1928 (time 1783951743.21 total 129.01 last 12.78) - open/close 2128 (time 1783951753.22 total 139.02 last 19.97) - open/close 2349 (time 1783951763.35 total 149.15 last 21.83) - open/close 2611 (time 1783951773.35 total 159.15 last 26.18) - open/close 2880 (time 1783951783.38 total 169.18 last 26.83) - open/close 3147 (time 1783951793.39 total 179.19 last 26.67) - open/close 3351 (time 1783951803.41 total 189.21 last 20.36) - open/close 3491 (time 1783951813.58 total 199.38 last 13.78) - open/close 3685 (time 1783951824.46 total 210.26 last 17.83) - open/close 3868 (time 1783951834.46 total 220.26 last 18.29) - open/close 4020 (time 1783951844.75 total 230.55 last 14.78) - open/close 4181 (time 1783951855.18 total 240.98 last 15.43) - open/close 4480 (time 1783951865.20 total 251.00 last 29.84) - open/close 4746 (time 1783951875.21 total 261.01 last 26.56) - open/close 5017 (time 1783951885.24 total 271.04 last 27.03) - open/close 5267 (time 1783951895.25 total 281.05 last 24.98) - open/close 5491 (time 1783951905.56 total 291.36 last 21.72) - open/close 5754 (time 1783951916.62 total 302.42 last 23.76) - open/close 6013 (time 1783951926.63 total 312.43 last 25.88) - open/close 6181 (time 1783951936.65 total 322.45 last 16.77) - open/close 6364 (time 1783951946.97 total 332.78 last 17.72) - open/close 6525 (time 1783951957.31 total 343.11 last 15.57) - open/close 6711 (time 1783951967.79 total 353.59 last 17.75) - open/close 6929 (time 1783951978.14 total 363.94 last 21.07) - open/close 7207 (time 1783951988.16 total 373.96 last 27.74) - open/close 7454 (time 1783952000.26 total 386.06 last 20.41) - open/close 7705 (time 1783952010.71 total 396.51 last 24.03) - open/close 7882 (time 1783952020.75 total 406.55 last 17.62) - open/close 8027 (time 1783952031.94 total 417.74 last 12.96) - open/close 8305 (time 1783952041.94 total 427.74 last 27.78) - open/close 8538 (time 1783952052.11 total 437.91 last 22.91) - open/close 8754 (time 1783952062.13 total 447.93 last 21.57) - open/close 8914 (time 1783952072.14 total 457.94 last 15.97) - open/close 9027 (time 1783952082.89 total 468.69 last 10.51) - open/close 9228 (time 1783952092.90 total 478.70 last 20.09) - open/close 9473 (time 1783952103.43 total 489.23 last 23.26) - open/close 9736 (time 1783952114.00 total 499.80 last 24.89) - open/close 9930 (time 1783952125.91 total 511.71 last 16.29) - open/close 10000 (time 1783952129.18 total 514.98 last 21.38) - open/close 10277 (time 1783952140.01 total 525.81 last 25.58) - open/close 10551 (time 1783952150.02 total 535.82 last 27.37) - open/close 10819 (time 1783952160.45 total 546.25 last 25.71) - open/close 11091 (time 1783952170.46 total 556.26 last 27.15) - open/close 11327 (time 1783952180.47 total 566.27 last 23.58) - open/close 11600 (time 1783952191.37 total 577.17 last 25.05) - open/close 11852 (time 1783952202.06 total 587.86 last 23.58) - open/close 12063 (time 1783952212.94 total 598.74 last 19.41) - open/close 12260 (time 1783952223.64 total 609.44 last 18.40) - open/close 12454 (time 1783952233.69 total 619.49 last 19.30) - open/close 12677 (time 1783952244.57 total 630.37 last 20.51) - open/close 12803 (time 1783952254.58 total 640.38 last 12.58) - open/close 12909 (time 1783952265.10 total 650.90 last 10.08) - open/close 12974 (time 1783952275.10 total 660.90 last 6.50) - open/close 13031 (time 1783952285.57 total 671.37 last 5.45) - open/close 13143 (time 1783952296.08 total 681.88 last 10.66) - open/close 13356 (time 1783952306.38 total 692.18 last 20.67) - open/close 13610 (time 1783952317.27 total 703.07 last 23.32) - open/close 13747 (time 1783952327.43 total 713.23 last 13.49) - open/close 14000 (time 1783952338.25 total 724.05 last 23.38) - open/close 14104 (time 1783952348.26 total 734.06 last 10.39) - open/close 14267 (time 1783952360.71 total 746.51 last 13.09) - open/close 14433 (time 1783952371.15 total 756.95 last 15.91) - open/close 14690 (time 1783952381.17 total 766.97 last 25.63) - open/close 14939 (time 1783952391.18 total 776.98 last 24.87) - open/close 15100 (time 1783952401.19 total 786.99 last 16.08) - open/close 15342 (time 1783952411.46 total 797.26 last 23.57) - open/close 15601 (time 1783952421.51 total 807.31 last 25.77) - open/close 15759 (time 1783952432.77 total 818.57 last 14.03) - open/close 16028 (time 1783952443.72 total 829.52 last 24.55) - open/close 16315 (time 1783952454.09 total 839.89 last 27.70) - open/close 16560 (time 1783952464.10 total 849.90 last 24.46) - open/close 16792 (time 1783952475.00 total 860.80 last 21.28) - open/close 16998 (time 1783952485.01 total 870.81 last 20.59) - open/close 17140 (time 1783952496.02 total 881.82 last 12.89) - open/close 17342 (time 1783952506.92 total 892.72 last 18.52) - open/close 17568 (time 1783952517.32 total 903.12 last 21.73) - open/close 17792 (time 1783952527.33 total 913.13 last 22.37) - open/close 17986 (time 1783952537.34 total 923.14 last 19.38) - open/close 18244 (time 1783952547.78 total 933.58 last 24.73) - open/close 18525 (time 1783952558.80 total 944.60 last 25.48) - open/close 18788 (time 1783952568.82 total 954.62 last 26.26) - open/close 19074 (time 1783952578.83 total 964.63 last 28.57) - open/close 19306 (time 1783952588.87 total 974.67 last 23.09) - open/close 19611 (time 1783952598.88 total 984.68 last 30.49) - open/close 20000 (time 1783952604.25 total 990.05 last 72.34) - open/close 20729 (time 1783952614.26 total 1000.06 last 72.87) - open/close 21531 (time 1783952624.27 total 1010.07 last 80.12) - open/close 22263 (time 1783952634.27 total 1020.07 last 73.19) - open/close 23036 (time 1783952644.29 total 1030.09 last 77.11) - open/close 23771 (time 1783952654.31 total 1040.11 last 73.38) - open/close 24378 (time 1783952664.32 total 1050.12 last 60.66) - open/close 25019 (time 1783952674.33 total 1060.13 last 64.02) - open/close 25747 (time 1783952684.33 total 1070.13 last 72.76) - open/close 26449 (time 1783952694.33 total 1080.13 last 70.19) - open/close 27153 (time 1783952704.34 total 1090.14 last 70.34) - open/close 27915 (time 1783952714.36 total 1100.16 last 76.06) - open/close 28633 (time 1783952724.36 total 1110.17 last 71.78) - open/close 29287 (time 1783952734.39 total 1120.19 last 65.24) - open/close 29916 (time 1783952744.40 total 1130.20 last 62.81) - open/close 30513 (time 1783952754.41 total 1140.21 last 59.67) - open/close 31179 (time 1783952764.42 total 1150.22 last 66.55) - open/close 31840 (time 1783952774.42 total 1160.22 last 66.07) - open/close 32555 (time 1783952784.43 total 1170.23 last 71.43) - open/close 33380 (time 1783952794.43 total 1180.23 last 82.48) - open/close 34215 (time 1783952804.44 total 1190.24 last 83.43) - open/close 34978 (time 1783952814.45 total 1200.25 last 76.24) - open/close 35772 (time 1783952824.46 total 1210.26 last 79.30) - open/close 36567 (time 1783952834.47 total 1220.27 last 79.44) - open/close 37372 (time 1783952844.47 total 1230.27 last 80.47) - open/close 38181 (time 1783952854.48 total 1240.28 last 80.88) - open/close 38902 (time 1783952864.48 total 1250.28 last 72.10) - open/close 39583 (time 1783952874.48 total 1260.28 last 68.07) - open/close 40000 (time 1783952880.08 total 1265.88 last 74.49) - open/close 40695 (time 1783952890.09 total 1275.89 last 69.44) - open/close 41494 (time 1783952900.09 total 1285.89 last 79.87) - open/close 42326 (time 1783952910.10 total 1295.90 last 83.09) - open/close 43131 (time 1783952920.11 total 1305.91 last 80.46) - open/close 43937 (time 1783952930.13 total 1315.93 last 80.47) - open/close 44751 (time 1783952940.14 total 1325.94 last 81.32) - open/close 45564 (time 1783952950.14 total 1335.94 last 81.28) - open/close 46294 (time 1783952960.15 total 1345.95 last 72.94) - open/close 47004 (time 1783952970.16 total 1355.96 last 70.90) - open/close 47775 (time 1783952980.17 total 1365.97 last 77.02) - open/close 48625 (time 1783952990.18 total 1375.98 last 84.97) - open/close 49385 (time 1783953000.18 total 1385.98 last 75.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1389.05 seconds: 35.73 ops/second - unlinked 0 (time 1783953007 ; total 0 ; last 0) - unlinked 10000 (time 1783953145 ; total 138 ; last 138) - unlinked 20000 (time 1783953275 ; total 268 ; last 130) - unlinked 30000 (time 1783953400 ; total 393 ; last 125) - unlinked 40000 (time 1783953536 ; total 529 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 658 seconds: 75.427055 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2