== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:36:50 (1787852210) start mds service on oleg604-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg604-server' oleg604-server: oleg604-server.virtnet: executing load_modules_local oleg604-server: Loading modules from /home/green/git/lustre-release/lustre oleg604-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg604-server: detected 4 online CPUs by sysfs oleg604-server: MODOPTS_LIBCFS= oleg604-server: Force libcfs to create 2 CPU partitions oleg604-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg604-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg604-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg604-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg604-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg604-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg604-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg604-server: oleg604-server.virtnet: executing set_default_debug -1 all pdsh@oleg604-client: oleg604-server: ssh exited with exit code 1 Started lustre-OST0000 oleg604-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg604-client.virtnet: -o user_xattr,flock 192.168.206.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1787852322.69 total 10.01 last 24.98) - open/close 410 (time 1787852332.69 total 20.01 last 15.99) - open/close 645 (time 1787852343.37 total 30.69 last 22.02) - open/close 889 (time 1787852354.76 total 42.08 last 21.40) - open/close 1052 (time 1787852365.31 total 52.63 last 15.45) - open/close 1249 (time 1787852376.22 total 63.54 last 18.06) - open/close 1383 (time 1787852386.23 total 73.55 last 13.38) - open/close 1614 (time 1787852396.33 total 83.65 last 22.87) - open/close 1770 (time 1787852406.61 total 93.94 last 15.17) - open/close 1950 (time 1787852416.84 total 104.16 last 17.60) - open/close 2189 (time 1787852427.78 total 115.10 last 21.85) - open/close 2343 (time 1787852440.11 total 127.43 last 12.49) - open/close 2531 (time 1787852450.88 total 138.20 last 17.45) - open/close 2721 (time 1787852460.89 total 148.21 last 18.99) - open/close 2904 (time 1787852470.92 total 158.24 last 18.24) - open/close 3099 (time 1787852480.93 total 168.25 last 19.48) - open/close 3314 (time 1787852491.90 total 179.22 last 19.59) - open/close 3461 (time 1787852502.89 total 190.21 last 13.38) - open/close 3635 (time 1787852513.24 total 200.56 last 16.80) - open/close 3744 (time 1787852524.00 total 211.32 last 10.13) - open/close 3913 (time 1787852534.87 total 222.19 last 15.54) - open/close 4086 (time 1787852544.87 total 232.19 last 17.30) - open/close 4250 (time 1787852555.53 total 242.85 last 15.38) - open/close 4413 (time 1787852566.31 total 253.63 last 15.13) - open/close 4551 (time 1787852577.24 total 264.56 last 12.62) - open/close 4757 (time 1787852587.78 total 275.10 last 19.56) - open/close 4954 (time 1787852598.49 total 285.81 last 18.39) - open/close 5129 (time 1787852608.65 total 295.97 last 17.23) - open/close 5373 (time 1787852619.20 total 306.52 last 23.12) - open/close 5547 (time 1787852629.22 total 316.54 last 17.37) - open/close 5744 (time 1787852639.77 total 327.09 last 18.67) - open/close 5913 (time 1787852650.28 total 337.60 last 16.08) - open/close 6152 (time 1787852660.30 total 347.62 last 23.84) - open/close 6276 (time 1787852670.69 total 358.01 last 11.94) - open/close 6496 (time 1787852681.15 total 368.47 last 21.04) - open/close 6585 (time 1787852691.16 total 378.48 last 8.89) - open/close 6712 (time 1787852702.13 total 389.45 last 11.58) - open/close 6933 (time 1787852712.14 total 399.46 last 22.10) - open/close 7123 (time 1787852722.14 total 409.46 last 18.98) - open/close 7271 (time 1787852732.31 total 419.63 last 14.56) - open/close 7540 (time 1787852743.04 total 430.36 last 25.06) - open/close 7808 (time 1787852753.23 total 440.55 last 26.31) - open/close 8028 (time 1787852763.97 total 451.29 last 20.48) - open/close 8233 (time 1787852774.15 total 461.47 last 20.13) - open/close 8453 (time 1787852784.15 total 471.47 last 22.00) - open/close 8659 (time 1787852794.51 total 481.83 last 19.90) - open/close 8885 (time 1787852804.61 total 491.93 last 22.37) - open/close 9074 (time 1787852817.38 total 504.70 last 14.80) - open/close 9196 (time 1787852827.38 total 514.70 last 12.19) - open/close 9307 (time 1787852838.16 total 525.48 last 10.30) - open/close 9559 (time 1787852849.13 total 536.45 last 22.99) - open/close 9821 (time 1787852859.91 total 547.23 last 24.30) - open/close 10000 (time 1787852868.49 total 555.81 last 20.87) - open/close 10202 (time 1787852879.25 total 566.57 last 18.76) - open/close 10471 (time 1787852889.27 total 576.59 last 26.85) - open/close 10731 (time 1787852899.40 total 586.72 last 25.66) - open/close 10938 (time 1787852910.27 total 597.59 last 19.06) - open/close 11197 (time 1787852920.41 total 607.73 last 25.53) - open/close 11451 (time 1787852930.65 total 617.97 last 24.82) - open/close 11736 (time 1787852941.38 total 628.70 last 26.54) - open/close 12007 (time 1787852951.63 total 638.95 last 26.45) - open/close 12263 (time 1787852961.85 total 649.17 last 25.04) - open/close 12516 (time 1787852971.85 total 659.17 last 25.30) - open/close 12701 (time 1787852981.91 total 669.23 last 18.40) - open/close 12854 (time 1787852991.91 total 679.23 last 15.29) - open/close 13072 (time 1787853001.93 total 689.25 last 21.75) - open/close 13239 (time 1787853011.94 total 699.26 last 16.68) - open/close 13478 (time 1787853021.95 total 709.27 last 23.88) - open/close 13697 (time 1787853032.34 total 719.66 last 21.08) - open/close 13962 (time 1787853042.37 total 729.69 last 26.41) - open/close 14177 (time 1787853052.43 total 739.75 last 21.37) - open/close 14308 (time 1787853062.46 total 749.78 last 13.07) - open/close 14507 (time 1787853072.48 total 759.80 last 19.86) - open/close 14704 (time 1787853083.48 total 770.80 last 17.91) - open/close 14916 (time 1787853093.51 total 780.83 last 21.12) - open/close 15173 (time 1787853103.53 total 790.85 last 25.66) - open/close 15419 (time 1787853113.53 total 800.85 last 24.59) - open/close 15604 (time 1787853123.54 total 810.86 last 18.48) - open/close 15802 (time 1787853133.73 total 821.05 last 19.44) - open/close 15955 (time 1787853144.22 total 831.54 last 14.59) - open/close 16168 (time 1787853155.21 total 842.53 last 19.38) - open/close 16427 (time 1787853165.91 total 853.23 last 24.22) - open/close 16656 (time 1787853177.23 total 864.55 last 20.22) - open/close 16962 (time 1787853187.25 total 874.57 last 30.54) - open/close 17207 (time 1787853197.26 total 884.58 last 24.48) - open/close 17413 (time 1787853207.27 total 894.59 last 20.57) - open/close 17676 (time 1787853218.25 total 905.57 last 23.96) - open/close 17907 (time 1787853228.82 total 916.14 last 21.86) - open/close 18062 (time 1787853238.83 total 926.15 last 15.48) - open/close 18337 (time 1787853249.28 total 936.60 last 26.33) - open/close 18622 (time 1787853260.31 total 947.63 last 25.83) - open/close 18890 (time 1787853270.92 total 958.24 last 25.26) - open/close 19099 (time 1787853280.92 total 968.25 last 20.89) - open/close 19515 (time 1787853290.93 total 978.25 last 41.57) - open/close 20000 (time 1787853297.76 total 985.08 last 71.05) - open/close 20660 (time 1787853307.76 total 995.08 last 65.96) - open/close 21358 (time 1787853317.79 total 1005.11 last 69.64) - open/close 21937 (time 1787853327.79 total 1015.11 last 57.86) - open/close 22723 (time 1787853337.79 total 1025.11 last 78.59) - open/close 23511 (time 1787853347.80 total 1035.12 last 78.77) - open/close 24249 (time 1787853357.80 total 1045.12 last 73.77) - open/close 24993 (time 1787853367.80 total 1055.12 last 74.38) - open/close 25791 (time 1787853377.81 total 1065.13 last 79.76) - open/close 26550 (time 1787853387.81 total 1075.13 last 75.88) - open/close 27320 (time 1787853397.82 total 1085.14 last 76.93) - open/close 28125 (time 1787853407.83 total 1095.15 last 80.40) - open/close 28880 (time 1787853417.84 total 1105.16 last 75.43) - open/close 29675 (time 1787853427.87 total 1115.19 last 79.32) - open/close 30000 (time 1787853431.17 total 1118.49 last 98.44) - open/close 30811 (time 1787853441.18 total 1128.50 last 80.98) - open/close 31611 (time 1787853451.20 total 1138.52 last 79.88) - open/close 32357 (time 1787853461.23 total 1148.55 last 74.38) - open/close 33136 (time 1787853471.24 total 1158.56 last 77.80) - open/close 33867 (time 1787853481.25 total 1168.57 last 72.99) - open/close 34577 (time 1787853491.25 total 1178.57 last 71.00) - open/close 35350 (time 1787853501.26 total 1188.58 last 77.22) - open/close 36112 (time 1787853511.28 total 1198.60 last 76.11) - open/close 36861 (time 1787853521.29 total 1208.61 last 74.82) - open/close 37651 (time 1787853531.30 total 1218.62 last 78.93) - open/close 38452 (time 1787853541.31 total 1228.64 last 79.94) - open/close 39336 (time 1787853551.32 total 1238.64 last 88.35) - open/close 40000 (time 1787853559.70 total 1247.02 last 79.25) - open/close 40797 (time 1787853569.70 total 1257.02 last 79.69) - open/close 41551 (time 1787853579.71 total 1267.03 last 75.36) - open/close 42368 (time 1787853589.71 total 1277.03 last 81.64) - open/close 43163 (time 1787853599.72 total 1287.04 last 79.47) - open/close 43922 (time 1787853609.72 total 1297.04 last 75.86) - open/close 44725 (time 1787853619.72 total 1307.04 last 80.29) - open/close 45553 (time 1787853629.73 total 1317.05 last 82.71) - open/close 46315 (time 1787853639.74 total 1327.06 last 76.18) - open/close 47077 (time 1787853649.75 total 1337.07 last 76.08) - open/close 47789 (time 1787853659.75 total 1347.07 last 71.20) - open/close 48579 (time 1787853669.76 total 1357.08 last 78.92) - open/close 49350 (time 1787853679.78 total 1367.10 last 76.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1371.07 seconds: 36.20 ops/second - unlinked 0 (time 1787853688 ; total 0 ; last 0) - unlinked 10000 (time 1787853812 ; total 124 ; last 124) - unlinked 20000 (time 1787853953 ; total 265 ; last 141) - unlinked 30000 (time 1787854078 ; total 390 ; last 125) - unlinked 40000 (time 1787854215 ; total 527 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 651 seconds: 76.238098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg604-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg604-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg604-server stop mds service on oleg604-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg604-server stop mds service on oleg604-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg604-server unloading modules via unload_modules_local on: 'oleg604-server' oleg604-server: oleg604-server.virtnet: executing unload_modules_local oleg604-server: modules unloaded. pdsh@oleg604-client: oleg604-client: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-server: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-client: ssh exited with exit code 2 pdsh@oleg604-client: oleg604-server: ssh exited with exit code 2