== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:18:54 (1788445134) start mds service on oleg305-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg305-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg305-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 171 (time 1788445230.56 total 13.03 last 13.12) - open/close 425 (time 1788445240.58 total 23.05 last 25.36) - open/close 609 (time 1788445250.59 total 33.06 last 18.38) - open/close 872 (time 1788445261.08 total 43.55 last 25.07) - open/close 1027 (time 1788445271.09 total 53.56 last 15.49) - open/close 1189 (time 1788445281.10 total 63.57 last 16.18) - open/close 1286 (time 1788445291.36 total 73.82 last 9.46) - open/close 1463 (time 1788445301.37 total 83.84 last 17.68) - open/close 1598 (time 1788445312.26 total 94.73 last 12.39) - open/close 1739 (time 1788445322.72 total 105.19 last 13.49) - open/close 1909 (time 1788445333.68 total 116.15 last 15.51) - open/close 2075 (time 1788445344.74 total 127.21 last 15.01) - open/close 2216 (time 1788445355.30 total 137.77 last 13.35) - open/close 2346 (time 1788445365.88 total 148.34 last 12.30) - open/close 2493 (time 1788445375.90 total 158.37 last 14.66) - open/close 2694 (time 1788445386.98 total 169.45 last 18.14) - open/close 2943 (time 1788445397.00 total 179.47 last 24.86) - open/close 3151 (time 1788445407.28 total 189.75 last 20.24) - open/close 3323 (time 1788445417.28 total 199.75 last 17.20) - open/close 3457 (time 1788445427.29 total 209.76 last 13.40) - open/close 3633 (time 1788445439.20 total 221.67 last 14.78) - open/close 3740 (time 1788445450.83 total 233.30 last 9.20) - open/close 3961 (time 1788445461.61 total 244.07 last 20.51) - open/close 4141 (time 1788445472.14 total 254.61 last 17.08) - open/close 4264 (time 1788445482.83 total 265.30 last 11.51) - open/close 4331 (time 1788445493.21 total 275.68 last 6.46) - open/close 4453 (time 1788445503.25 total 285.72 last 12.14) - open/close 4680 (time 1788445514.10 total 296.57 last 20.92) - open/close 4873 (time 1788445524.11 total 306.58 last 19.28) - open/close 5104 (time 1788445535.00 total 317.47 last 21.22) - open/close 5333 (time 1788445545.01 total 327.47 last 22.89) - open/close 5560 (time 1788445555.54 total 338.01 last 21.55) - open/close 5751 (time 1788445565.85 total 348.32 last 18.53) - open/close 5926 (time 1788445575.89 total 358.35 last 17.43) - open/close 6062 (time 1788445586.21 total 368.68 last 13.17) - open/close 6226 (time 1788445596.72 total 379.18 last 15.61) - open/close 6323 (time 1788445606.74 total 389.21 last 9.68) - open/close 6470 (time 1788445616.77 total 399.24 last 14.65) - open/close 6615 (time 1788445629.73 total 412.19 last 11.20) - open/close 6807 (time 1788445639.73 total 422.19 last 19.20) - open/close 6942 (time 1788445651.69 total 434.16 last 11.28) - open/close 7148 (time 1788445662.50 total 444.97 last 19.06) - open/close 7300 (time 1788445675.44 total 457.91 last 11.75) - open/close 7469 (time 1788445685.54 total 468.01 last 16.73) - open/close 7689 (time 1788445695.55 total 478.02 last 21.98) - open/close 7803 (time 1788445705.55 total 488.02 last 11.40) - open/close 7945 (time 1788445716.46 total 498.93 last 13.02) - open/close 8067 (time 1788445727.31 total 509.78 last 11.25) - open/close 8184 (time 1788445737.31 total 519.78 last 11.69) - open/close 8456 (time 1788445747.33 total 529.80 last 27.15) - open/close 8674 (time 1788445757.35 total 539.81 last 21.77) - open/close 8877 (time 1788445768.17 total 550.64 last 18.75) - open/close 9032 (time 1788445781.14 total 563.61 last 11.95) - open/close 9195 (time 1788445791.16 total 573.63 last 16.27) - open/close 9302 (time 1788445801.89 total 584.36 last 9.97) - open/close 9487 (time 1788445811.93 total 594.39 last 18.43) - open/close 9670 (time 1788445821.93 total 604.40 last 18.29) - open/close 9867 (time 1788445831.94 total 614.41 last 19.68) - open/close 10000 (time 1788445840.16 total 622.63 last 16.18) - open/close 10222 (time 1788445850.16 total 632.63 last 22.19) - open/close 10363 (time 1788445861.20 total 643.67 last 12.78) - open/close 10533 (time 1788445872.03 total 654.50 last 15.69) - open/close 10757 (time 1788445883.03 total 665.50 last 20.37) - open/close 10951 (time 1788445893.03 total 675.50 last 19.39) - open/close 11186 (time 1788445903.12 total 685.59 last 23.30) - open/close 11337 (time 1788445913.15 total 695.61 last 15.06) - open/close 11443 (time 1788445923.15 total 705.62 last 10.60) - open/close 11585 (time 1788445933.74 total 716.21 last 13.40) - open/close 11746 (time 1788445943.91 total 726.38 last 15.84) - open/close 11934 (time 1788445954.59 total 737.06 last 17.61) - open/close 12080 (time 1788445966.82 total 749.28 last 11.94) - open/close 12281 (time 1788445977.12 total 759.59 last 19.51) - open/close 12543 (time 1788445987.12 total 769.59 last 26.19) - open/close 12729 (time 1788445997.14 total 779.61 last 18.57) - open/close 12858 (time 1788446007.21 total 789.68 last 12.80) - open/close 12961 (time 1788446018.21 total 800.68 last 9.37) - open/close 13188 (time 1788446028.37 total 810.84 last 22.35) - open/close 13290 (time 1788446039.00 total 821.47 last 9.59) - open/close 13508 (time 1788446049.03 total 831.49 last 21.75) - open/close 13622 (time 1788446059.04 total 841.50 last 11.39) - open/close 13704 (time 1788446069.08 total 851.55 last 8.16) - open/close 13849 (time 1788446079.10 total 861.57 last 14.47) - open/close 14008 (time 1788446090.03 total 872.50 last 14.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14066) error: No space left on device total: 14066 open/close in 877.14 seconds: 16.04 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76115) - No such process - unlinked 0 (time 1788446103 ; total 0 ; last 0) - unlinked 10000 (time 1788446250 ; total 147 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14066) error: No such file or directory total: 14066 unlinks in 188 seconds: 74.819145 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local oleg305-server: modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2