== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:56:50 (1784048210) start mds service on oleg427-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 213 (time 1784048284.23 total 10.32 last 20.65) - open/close 331 (time 1784048294.68 total 20.76 last 11.29) - open/close 561 (time 1784048305.69 total 31.78 last 20.89) - open/close 769 (time 1784048316.70 total 42.78 last 18.90) - open/close 1037 (time 1784048327.33 total 53.42 last 25.20) - open/close 1229 (time 1784048340.25 total 66.34 last 14.86) - open/close 1507 (time 1784048350.26 total 76.34 last 27.79) - open/close 1735 (time 1784048361.29 total 87.38 last 20.66) - open/close 2014 (time 1784048371.30 total 97.39 last 27.88) - open/close 2249 (time 1784048381.53 total 107.62 last 22.97) - open/close 2449 (time 1784048391.58 total 117.67 last 19.89) - open/close 2638 (time 1784048402.01 total 128.10 last 18.13) - open/close 2901 (time 1784048412.82 total 138.91 last 24.33) - open/close 3136 (time 1784048423.40 total 149.48 last 22.22) - open/close 3309 (time 1784048433.60 total 159.69 last 16.95) - open/close 3562 (time 1784048444.38 total 170.47 last 23.47) - open/close 3805 (time 1784048454.39 total 180.48 last 24.27) - open/close 4035 (time 1784048465.12 total 191.20 last 21.44) - open/close 4273 (time 1784048476.17 total 202.26 last 21.52) - open/close 4478 (time 1784048486.18 total 212.26 last 20.50) - open/close 4716 (time 1784048496.98 total 223.07 last 22.03) - open/close 4840 (time 1784048507.02 total 233.10 last 12.36) - open/close 5023 (time 1784048517.02 total 243.11 last 18.29) - open/close 5240 (time 1784048527.57 total 253.66 last 20.57) - open/close 5498 (time 1784048538.45 total 264.54 last 23.70) - open/close 5656 (time 1784048548.48 total 274.57 last 15.76) - open/close 5841 (time 1784048559.26 total 285.35 last 17.16) - open/close 6032 (time 1784048569.86 total 295.95 last 18.02) - open/close 6327 (time 1784048580.50 total 306.58 last 27.74) - open/close 6525 (time 1784048591.62 total 317.71 last 17.79) - open/close 6749 (time 1784048601.88 total 327.96 last 21.85) - open/close 6966 (time 1784048613.15 total 339.24 last 19.24) - open/close 7064 (time 1784048623.89 total 349.98 last 9.13) - open/close 7259 (time 1784048634.91 total 360.99 last 17.70) - open/close 7493 (time 1784048644.91 total 371.00 last 23.39) - open/close 7667 (time 1784048656.04 total 382.12 last 15.64) - open/close 7802 (time 1784048666.05 total 392.13 last 13.49) - open/close 8045 (time 1784048676.05 total 402.14 last 24.28) - open/close 8296 (time 1784048686.99 total 413.07 last 22.96) - open/close 8512 (time 1784048697.82 total 423.91 last 19.94) - open/close 8757 (time 1784048708.19 total 434.27 last 23.63) - open/close 9004 (time 1784048718.83 total 444.92 last 23.20) - open/close 9229 (time 1784048728.99 total 455.08 last 22.15) - open/close 9413 (time 1784048739.60 total 465.68 last 17.35) - open/close 9574 (time 1784048749.91 total 476.00 last 15.61) - open/close 9833 (time 1784048759.93 total 486.01 last 25.86) - open/close 10000 (time 1784048764.63 total 490.71 last 35.53) - open/close 10229 (time 1784048775.12 total 501.21 last 21.82) - open/close 10482 (time 1784048785.31 total 511.40 last 24.84) - open/close 10710 (time 1784048795.34 total 521.42 last 22.74) - open/close 10878 (time 1784048805.80 total 531.88 last 16.06) - open/close 11012 (time 1784048815.81 total 541.90 last 13.38) - open/close 11180 (time 1784048825.86 total 551.95 last 16.72) - open/close 11502 (time 1784048835.87 total 561.95 last 32.18) - open/close 11731 (time 1784048846.23 total 572.32 last 22.10) - open/close 11950 (time 1784048856.46 total 582.54 last 21.41) - open/close 12191 (time 1784048867.05 total 593.13 last 22.76) - open/close 12410 (time 1784048879.18 total 605.27 last 18.05) - open/close 12684 (time 1784048889.24 total 615.32 last 27.24) - open/close 12914 (time 1784048899.92 total 626.00 last 21.54) - open/close 13166 (time 1784048909.93 total 636.01 last 25.18) - open/close 13387 (time 1784048920.24 total 646.33 last 21.43) - open/close 13613 (time 1784048930.83 total 656.91 last 21.35) - open/close 13810 (time 1784048940.90 total 666.98 last 19.56) - open/close 14047 (time 1784048953.93 total 680.02 last 18.18) - open/close 14307 (time 1784048963.95 total 690.03 last 25.97) - open/close 14515 (time 1784048973.95 total 700.04 last 20.79) - open/close 14725 (time 1784048983.97 total 710.05 last 20.97) - open/close 15002 (time 1784048994.78 total 720.87 last 25.61) - open/close 15237 (time 1784049005.84 total 731.93 last 21.25) - open/close 15480 (time 1784049015.95 total 742.04 last 24.03) - open/close 15739 (time 1784049026.38 total 752.46 last 24.85) - open/close 15922 (time 1784049036.83 total 762.91 last 17.51) - open/close 16160 (time 1784049047.25 total 773.33 last 22.84) - open/close 16393 (time 1784049059.55 total 785.64 last 18.93) - open/close 16680 (time 1784049069.56 total 795.64 last 28.70) - open/close 16906 (time 1784049079.57 total 805.66 last 22.56) - open/close 17124 (time 1784049089.74 total 815.83 last 21.44) - open/close 17352 (time 1784049099.80 total 825.89 last 22.67) - open/close 17589 (time 1784049110.30 total 836.38 last 22.59) - open/close 17827 (time 1784049120.49 total 846.58 last 23.34) - open/close 18082 (time 1784049130.51 total 856.60 last 25.45) - open/close 18323 (time 1784049141.09 total 867.17 last 22.80) - open/close 18584 (time 1784049151.95 total 878.04 last 24.02) - open/close 18812 (time 1784049162.82 total 888.91 last 20.97) - open/close 19040 (time 1784049173.59 total 899.68 last 21.17) - open/close 19564 (time 1784049183.60 total 909.69 last 52.35) - open/close 20000 (time 1784049189.94 total 916.03 last 68.79) - open/close 20627 (time 1784049199.96 total 926.04 last 62.60) - open/close 21299 (time 1784049209.97 total 936.05 last 67.14) - open/close 22025 (time 1784049219.99 total 946.08 last 72.40) - open/close 22736 (time 1784049229.99 total 956.08 last 71.10) - open/close 23465 (time 1784049240.00 total 966.09 last 72.82) - open/close 24086 (time 1784049250.01 total 976.09 last 62.08) - open/close 24702 (time 1784049260.01 total 986.10 last 61.57) - open/close 25332 (time 1784049270.03 total 996.12 last 62.86) - open/close 25892 (time 1784049280.04 total 1006.13 last 55.94) - open/close 26475 (time 1784049290.05 total 1016.13 last 58.30) - open/close 27022 (time 1784049300.05 total 1026.14 last 54.66) - open/close 27555 (time 1784049310.06 total 1036.15 last 53.26) - open/close 28122 (time 1784049320.06 total 1046.15 last 56.69) - open/close 28687 (time 1784049330.09 total 1056.18 last 56.34) - open/close 29240 (time 1784049340.09 total 1066.18 last 55.29) - open/close 29848 (time 1784049350.10 total 1076.19 last 60.74) - open/close 30000 (time 1784049352.45 total 1078.53 last 64.83) - open/close 30610 (time 1784049362.45 total 1088.54 last 60.98) - open/close 31186 (time 1784049372.45 total 1098.54 last 57.58) - open/close 31796 (time 1784049382.46 total 1108.54 last 60.99) - open/close 32815 (time 1784049392.46 total 1118.55 last 101.84) - open/close 33560 (time 1784049402.47 total 1128.56 last 74.43) - open/close 34029 (time 1784049412.49 total 1138.57 last 46.82) - open/close 34409 (time 1784049422.49 total 1148.58 last 37.99) - open/close 34898 (time 1784049432.54 total 1158.63 last 48.65) - open/close 35503 (time 1784049442.54 total 1168.63 last 60.50) - open/close 36107 (time 1784049452.55 total 1178.64 last 60.34) - open/close 36787 (time 1784049462.56 total 1188.64 last 67.98) - open/close 37467 (time 1784049472.56 total 1198.64 last 68.00) - open/close 38136 (time 1784049482.57 total 1208.65 last 66.85) - open/close 38773 (time 1784049492.57 total 1218.66 last 63.66) - open/close 39484 (time 1784049502.58 total 1228.67 last 71.03) - open/close 40000 (time 1784049510.32 total 1236.40 last 66.71) - open/close 40630 (time 1784049520.33 total 1246.41 last 62.93) - open/close 41294 (time 1784049530.33 total 1256.42 last 66.37) - open/close 41937 (time 1784049540.34 total 1266.42 last 64.27) - open/close 42647 (time 1784049550.35 total 1276.44 last 70.90) - open/close 43345 (time 1784049560.36 total 1286.44 last 69.75) - open/close 44069 (time 1784049570.37 total 1296.46 last 72.31) - open/close 44734 (time 1784049580.38 total 1306.47 last 66.44) - open/close 45394 (time 1784049590.43 total 1316.51 last 65.71) - open/close 45996 (time 1784049600.43 total 1326.51 last 60.18) - open/close 46593 (time 1784049610.44 total 1336.53 last 59.60) - open/close 47279 (time 1784049620.45 total 1346.54 last 68.56) - open/close 47951 (time 1784049630.46 total 1356.55 last 67.12) - open/close 48618 (time 1784049640.47 total 1366.56 last 66.64) - open/close 49310 (time 1784049650.48 total 1376.56 last 69.15) total: 50000 open/close in 1385.87 seconds: 36.08 ops/second - unlinked 0 (time 1784049663 ; total 0 ; last 0) - unlinked 10000 (time 1784049819 ; total 156 ; last 156) - unlinked 20000 (time 1784050000 ; total 337 ; last 181) - unlinked 30000 (time 1784050173 ; total 510 ; last 173) - unlinked 40000 (time 1784050331 ; total 668 ; last 158) total: 50000 unlinks in 827 seconds: 60.459492 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2