== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:59:50 (1763513990) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 184 (time 1763514082.80 total 10.25 last 17.96) - open/close 376 (time 1763514092.93 total 20.38 last 18.95) - open/close 599 (time 1763514102.95 total 30.39 last 22.27) - open/close 830 (time 1763514113.92 total 41.37 last 21.04) - open/close 1072 (time 1763514123.93 total 51.38 last 24.17) - open/close 1296 (time 1763514134.08 total 61.53 last 22.07) - open/close 1566 (time 1763514145.14 total 72.59 last 24.43) - open/close 1809 (time 1763514155.58 total 83.03 last 23.26) - open/close 2022 (time 1763514166.61 total 94.06 last 19.32) - open/close 2173 (time 1763514176.70 total 104.14 last 14.97) - open/close 2371 (time 1763514187.31 total 114.76 last 18.65) - open/close 2597 (time 1763514197.63 total 125.08 last 21.90) - open/close 2895 (time 1763514207.65 total 135.10 last 29.75) - open/close 3113 (time 1763514217.81 total 145.26 last 21.46) - open/close 3303 (time 1763514227.89 total 155.34 last 18.84) - open/close 3519 (time 1763514238.40 total 165.85 last 20.55) - open/close 3772 (time 1763514249.14 total 176.59 last 23.56) - open/close 4031 (time 1763514259.78 total 187.23 last 24.34) - open/close 4230 (time 1763514269.81 total 197.26 last 19.85) - open/close 4418 (time 1763514280.26 total 207.71 last 17.98) - open/close 4654 (time 1763514291.00 total 218.45 last 21.97) - open/close 4887 (time 1763514301.95 total 229.40 last 21.29) - open/close 5107 (time 1763514313.01 total 240.46 last 19.88) - open/close 5339 (time 1763514323.90 total 251.35 last 21.31) - open/close 5461 (time 1763514334.39 total 261.84 last 11.63) - open/close 5621 (time 1763514345.20 total 272.65 last 14.79) - open/close 5832 (time 1763514355.24 total 282.69 last 21.03) - open/close 6092 (time 1763514365.24 total 292.69 last 26.00) - open/close 6294 (time 1763514375.47 total 302.92 last 19.74) - open/close 6511 (time 1763514385.49 total 312.94 last 21.66) - open/close 6741 (time 1763514395.50 total 322.95 last 22.98) - open/close 6961 (time 1763514405.50 total 332.95 last 21.99) - open/close 7183 (time 1763514415.52 total 342.97 last 22.17) - open/close 7369 (time 1763514425.90 total 353.35 last 17.91) - open/close 7526 (time 1763514436.59 total 364.04 last 14.68) - open/close 7682 (time 1763514447.28 total 374.73 last 14.59) - open/close 7860 (time 1763514457.39 total 384.84 last 17.61) - open/close 8047 (time 1763514467.51 total 394.96 last 18.47) - open/close 8229 (time 1763514477.75 total 405.20 last 17.78) - open/close 8406 (time 1763514487.80 total 415.25 last 17.62) - open/close 8580 (time 1763514498.74 total 426.18 last 15.91) - open/close 8769 (time 1763514509.07 total 436.51 last 18.30) - open/close 8947 (time 1763514519.09 total 446.54 last 17.76) - open/close 9177 (time 1763514530.04 total 457.49 last 21.00) - open/close 9346 (time 1763514540.79 total 468.23 last 15.73) - open/close 9553 (time 1763514550.80 total 478.25 last 20.67) - open/close 9781 (time 1763514560.80 total 488.25 last 22.79) - open/close 9950 (time 1763514571.70 total 499.14 last 15.51) - open/close 10000 (time 1763514574.46 total 501.91 last 18.07) - open/close 10183 (time 1763514585.40 total 512.85 last 16.73) - open/close 10299 (time 1763514595.63 total 523.08 last 11.33) - open/close 10448 (time 1763514606.07 total 533.52 last 14.27) - open/close 10680 (time 1763514616.09 total 543.54 last 23.16) - open/close 10927 (time 1763514626.10 total 553.55 last 24.68) - open/close 11172 (time 1763514636.11 total 563.56 last 24.48) - open/close 11383 (time 1763514646.11 total 573.56 last 21.09) - open/close 11627 (time 1763514657.03 total 584.47 last 22.36) - open/close 11885 (time 1763514667.65 total 595.10 last 24.28) - open/close 12131 (time 1763514678.27 total 605.72 last 23.17) - open/close 12342 (time 1763514688.97 total 616.42 last 19.72) - open/close 12555 (time 1763514699.91 total 627.36 last 19.46) - open/close 12729 (time 1763514710.90 total 638.35 last 15.83) - open/close 12938 (time 1763514720.91 total 648.35 last 20.90) - open/close 13153 (time 1763514730.92 total 658.37 last 21.47) - open/close 13355 (time 1763514741.68 total 669.13 last 18.77) - open/close 13603 (time 1763514752.42 total 679.87 last 23.09) - open/close 13810 (time 1763514762.44 total 689.88 last 20.68) - open/close 14022 (time 1763514772.44 total 699.89 last 21.19) - open/close 14222 (time 1763514782.45 total 709.90 last 19.98) - open/close 14438 (time 1763514792.46 total 719.90 last 21.58) - open/close 14638 (time 1763514803.32 total 730.77 last 18.41) - open/close 14859 (time 1763514814.26 total 741.71 last 20.20) - open/close 15097 (time 1763514825.27 total 752.72 last 21.63) - open/close 15320 (time 1763514835.27 total 762.72 last 22.30) - open/close 15549 (time 1763514845.75 total 773.20 last 21.85) - open/close 15819 (time 1763514856.76 total 784.21 last 24.52) - open/close 16437 (time 1763514866.77 total 794.22 last 61.75) - open/close 17283 (time 1763514876.77 total 804.22 last 84.57) - open/close 18132 (time 1763514886.79 total 814.23 last 84.80) - open/close 18935 (time 1763514896.80 total 824.25 last 80.16) - open/close 19756 (time 1763514906.80 total 834.25 last 82.09) - open/close 20000 (time 1763514909.77 total 837.22 last 82.24) - open/close 20889 (time 1763514919.78 total 847.23 last 88.79) - open/close 21758 (time 1763514929.79 total 857.24 last 86.83) - open/close 22644 (time 1763514939.79 total 867.24 last 88.58) - open/close 23578 (time 1763514949.81 total 877.25 last 93.30) - open/close 24542 (time 1763514959.81 total 887.26 last 96.39) - open/close 25404 (time 1763514969.82 total 897.27 last 86.09) - open/close 26209 (time 1763514979.83 total 907.28 last 80.40) - open/close 27065 (time 1763514989.83 total 917.28 last 85.60) - open/close 27860 (time 1763514999.84 total 927.29 last 79.47) - open/close 28652 (time 1763515009.85 total 937.30 last 79.10) - open/close 29447 (time 1763515019.85 total 947.30 last 79.45) - open/close 30000 (time 1763515026.22 total 953.67 last 86.92) - open/close 30847 (time 1763515036.22 total 963.67 last 84.70) - open/close 31712 (time 1763515046.23 total 973.68 last 86.41) - open/close 32600 (time 1763515056.24 total 983.69 last 88.68) - open/close 33464 (time 1763515066.25 total 993.70 last 86.34) - open/close 34263 (time 1763515076.25 total 1003.70 last 79.86) - open/close 35106 (time 1763515086.26 total 1013.71 last 84.26) - open/close 35941 (time 1763515096.26 total 1023.71 last 83.50) - open/close 36802 (time 1763515106.26 total 1033.71 last 86.06) - open/close 37619 (time 1763515116.26 total 1043.71 last 81.70) - open/close 38436 (time 1763515126.26 total 1053.71 last 81.70) - open/close 39344 (time 1763515136.27 total 1063.72 last 90.73) - open/close 40000 (time 1763515142.99 total 1070.44 last 97.70) - open/close 40927 (time 1763515152.99 total 1080.44 last 92.66) - open/close 41911 (time 1763515163.00 total 1090.45 last 98.34) - open/close 42766 (time 1763515173.01 total 1100.46 last 85.39) - open/close 43653 (time 1763515183.01 total 1110.46 last 88.66) - open/close 44566 (time 1763515193.02 total 1120.47 last 91.25) - open/close 45432 (time 1763515203.03 total 1130.48 last 86.49) - open/close 46251 (time 1763515213.03 total 1140.48 last 81.88) - open/close 47093 (time 1763515223.03 total 1150.48 last 84.19) - open/close 47926 (time 1763515233.05 total 1160.49 last 83.21) - open/close 48877 (time 1763515243.05 total 1170.50 last 95.04) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1178.65 seconds: 42.11 ops/second - unlinked 0 (time 1763515253 ; total 0 ; last 0) - unlinked 10000 (time 1763515368 ; total 115 ; last 115) - unlinked 20000 (time 1763515460 ; total 207 ; last 92) - unlinked 30000 (time 1763515523 ; total 270 ; last 63) - unlinked 40000 (time 1763515587 ; total 334 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 392 seconds: 126.612244 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2