== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:59:09 (1787399949) start mds service on oleg336-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg336-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1787400065.55 total 10.75 last 14.79) - open/close 298 (time 1787400075.56 total 20.76 last 13.89) - open/close 520 (time 1787400085.58 total 30.78 last 22.16) - open/close 798 (time 1787400095.59 total 40.79 last 27.77) - open/close 1097 (time 1787400105.60 total 50.81 last 29.85) - open/close 1337 (time 1787400115.73 total 60.94 last 23.69) - open/close 1530 (time 1787400126.24 total 71.44 last 18.38) - open/close 1771 (time 1787400136.60 total 81.81 last 23.25) - open/close 2037 (time 1787400146.63 total 91.83 last 26.54) - open/close 2297 (time 1787400157.08 total 102.28 last 24.88) - open/close 2568 (time 1787400167.80 total 113.00 last 25.27) - open/close 2867 (time 1787400178.80 total 124.00 last 27.19) - open/close 3138 (time 1787400189.27 total 134.47 last 25.88) - open/close 3450 (time 1787400199.28 total 144.48 last 31.17) - open/close 3670 (time 1787400209.28 total 154.48 last 22.00) - open/close 3957 (time 1787400219.92 total 165.12 last 26.98) - open/close 4219 (time 1787400230.50 total 175.70 last 24.76) - open/close 4475 (time 1787400240.50 total 185.70 last 25.60) - open/close 4711 (time 1787400251.52 total 196.72 last 21.42) - open/close 5043 (time 1787400261.52 total 206.72 last 33.18) - open/close 5266 (time 1787400272.34 total 217.55 last 20.60) - open/close 5556 (time 1787400282.35 total 227.55 last 28.99) - open/close 5810 (time 1787400292.35 total 237.55 last 25.40) - open/close 5992 (time 1787400302.37 total 247.57 last 18.17) - open/close 6273 (time 1787400312.38 total 257.58 last 28.07) - open/close 6494 (time 1787400322.41 total 267.61 last 22.04) - open/close 6715 (time 1787400332.57 total 277.77 last 21.76) - open/close 6857 (time 1787400342.95 total 288.15 last 13.67) - open/close 7025 (time 1787400352.97 total 298.17 last 16.77) - open/close 7222 (time 1787400362.98 total 308.19 last 19.67) - open/close 7484 (time 1787400373.58 total 318.78 last 24.73) - open/close 7759 (time 1787400383.58 total 328.79 last 27.49) - open/close 8065 (time 1787400393.59 total 338.79 last 30.59) - open/close 8314 (time 1787400403.73 total 348.93 last 24.54) - open/close 8520 (time 1787400413.74 total 358.94 last 20.59) - open/close 8795 (time 1787400426.01 total 371.21 last 22.40) - open/close 8980 (time 1787400436.69 total 381.89 last 17.32) - open/close 9250 (time 1787400447.29 total 392.49 last 25.47) - open/close 9428 (time 1787400457.30 total 402.50 last 17.79) - open/close 9505 (time 1787400469.72 total 414.92 last 6.20) - open/close 9710 (time 1787400479.72 total 424.92 last 20.49) - open/close 9889 (time 1787400490.59 total 435.79 last 16.47) - open/close 10000 (time 1787400494.41 total 439.61 last 29.02) - open/close 10271 (time 1787400504.42 total 449.62 last 27.07) - open/close 10465 (time 1787400514.42 total 459.63 last 19.40) - open/close 10716 (time 1787400525.35 total 470.55 last 22.98) - open/close 10973 (time 1787400535.36 total 480.56 last 25.66) - open/close 11170 (time 1787400546.51 total 491.72 last 17.67) - open/close 11444 (time 1787400556.54 total 501.74 last 27.34) - open/close 11681 (time 1787400566.75 total 511.95 last 23.20) - open/close 11950 (time 1787400577.04 total 522.24 last 26.14) - open/close 12223 (time 1787400587.75 total 532.95 last 25.49) - open/close 12458 (time 1787400597.77 total 542.97 last 23.46) - open/close 12660 (time 1787400608.08 total 553.28 last 19.59) - open/close 12848 (time 1787400618.09 total 563.29 last 18.79) - open/close 13078 (time 1787400628.09 total 573.29 last 23.00) - open/close 13309 (time 1787400638.83 total 584.03 last 21.51) - open/close 13548 (time 1787400648.84 total 594.04 last 23.87) - open/close 13823 (time 1787400658.84 total 604.04 last 27.49) - open/close 14074 (time 1787400668.85 total 614.06 last 25.07) - open/close 14283 (time 1787400678.86 total 624.06 last 20.90) - open/close 14599 (time 1787400689.88 total 635.08 last 28.67) - open/close 14885 (time 1787400700.78 total 645.98 last 26.24) - open/close 15135 (time 1787400710.82 total 656.02 last 24.90) - open/close 15391 (time 1787400720.86 total 666.07 last 25.48) - open/close 15643 (time 1787400731.22 total 676.42 last 24.34) - open/close 15837 (time 1787400741.39 total 686.59 last 19.07) - open/close 16049 (time 1787400751.69 total 696.90 last 20.58) - open/close 16365 (time 1787400762.44 total 707.64 last 29.40) - open/close 16610 (time 1787400772.46 total 717.66 last 24.46) - open/close 16699 (time 1787400782.46 total 727.66 last 8.90) - open/close 16806 (time 1787400794.59 total 739.79 last 8.82) - open/close 17085 (time 1787400805.17 total 750.38 last 26.36) - open/close 17330 (time 1787400815.19 total 760.39 last 24.46) - open/close 17590 (time 1787400825.30 total 770.50 last 25.72) - open/close 17851 (time 1787400835.54 total 780.75 last 25.48) - open/close 18097 (time 1787400846.37 total 791.57 last 22.72) - open/close 18272 (time 1787400857.29 total 802.50 last 16.02) - open/close 18368 (time 1787400867.31 total 812.51 last 9.59) - open/close 18524 (time 1787400877.32 total 822.52 last 15.57) - open/close 18707 (time 1787400887.35 total 832.55 last 18.25) - open/close 18953 (time 1787400897.36 total 842.56 last 24.58) - open/close 19258 (time 1787400908.24 total 853.45 last 28.02) - open/close 19553 (time 1787400919.06 total 864.26 last 27.27) - open/close 19678 (time 1787400929.55 total 874.75 last 11.92) - open/close 19941 (time 1787400939.57 total 884.77 last 26.25) - open/close 20000 (time 1787400942.68 total 887.88 last 18.97) - open/close 20234 (time 1787400952.73 total 897.93 last 23.28) - open/close 20506 (time 1787400962.74 total 907.94 last 27.18) - open/close 20746 (time 1787400972.75 total 917.95 last 23.97) - open/close 21147 (time 1787400982.75 total 927.95 last 40.10) - open/close 22059 (time 1787400992.76 total 937.96 last 91.10) - open/close 22902 (time 1787401002.76 total 947.97 last 84.28) - open/close 23900 (time 1787401012.77 total 957.97 last 99.72) - open/close 24626 (time 1787401022.77 total 967.98 last 72.59) - open/close 25443 (time 1787401032.78 total 977.98 last 81.67) - open/close 26251 (time 1787401042.78 total 987.98 last 80.77) - open/close 27181 (time 1787401052.79 total 997.99 last 92.92) - open/close 28015 (time 1787401062.79 total 1007.99 last 83.39) - open/close 28920 (time 1787401072.80 total 1018.00 last 90.44) - open/close 29778 (time 1787401082.80 total 1028.00 last 85.78) - open/close 30000 (time 1787401085.10 total 1030.31 last 96.32) - open/close 30873 (time 1787401095.11 total 1040.31 last 87.25) - open/close 31794 (time 1787401105.12 total 1050.32 last 92.04) - open/close 32672 (time 1787401115.12 total 1060.32 last 87.79) - open/close 33562 (time 1787401125.13 total 1070.33 last 88.92) - open/close 34437 (time 1787401135.13 total 1080.33 last 87.49) - open/close 35287 (time 1787401145.13 total 1090.34 last 84.95) - open/close 36047 (time 1787401155.14 total 1100.34 last 75.96) - open/close 36874 (time 1787401165.15 total 1110.35 last 82.65) - open/close 37723 (time 1787401175.15 total 1120.35 last 84.87) - open/close 38545 (time 1787401185.15 total 1130.35 last 82.20) - open/close 39407 (time 1787401195.16 total 1140.36 last 86.09) - open/close 40000 (time 1787401201.72 total 1146.92 last 90.41) - open/close 40806 (time 1787401211.73 total 1156.93 last 80.57) - open/close 41592 (time 1787401221.73 total 1166.94 last 78.54) - open/close 42461 (time 1787401231.75 total 1176.95 last 86.73) - open/close 43311 (time 1787401241.76 total 1186.96 last 84.92) - open/close 44187 (time 1787401251.77 total 1196.97 last 87.56) - open/close 45084 (time 1787401261.77 total 1206.97 last 89.69) - open/close 45964 (time 1787401271.77 total 1216.97 last 87.95) - open/close 46794 (time 1787401281.77 total 1226.98 last 83.00) - open/close 47509 (time 1787401291.79 total 1236.99 last 71.40) - open/close 48348 (time 1787401301.79 total 1246.99 last 83.85) - open/close 49226 (time 1787401311.80 total 1257.00 last 87.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1261.24 seconds: 39.35 ops/second - unlinked 0 (time 1787401319 ; total 0 ; last 0) - unlinked 10000 (time 1787401451 ; total 132 ; last 132) - unlinked 20000 (time 1787401584 ; total 265 ; last 133) - unlinked 30000 (time 1787401711 ; total 392 ; last 127) - unlinked 40000 (time 1787401836 ; total 517 ; last 125) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 647 seconds: 76.709427 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2