== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:04:48 (1762491888) start mds service on oleg622-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing load_modules_local oleg622-server: Loading modules from /home/green/git/lustre-release/lustre oleg622-server: detected 4 online CPUs by sysfs oleg622-server: MODOPTS_LIBCFS= oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg622-server: Force libcfs to create 2 CPU partitions oleg622-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg622-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg622-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg622-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1762491952.96 total 10.67 last 18.27) - open/close 404 (time 1762491963.96 total 21.67 last 19.00) - open/close 587 (time 1762491974.57 total 32.28 last 17.25) - open/close 793 (time 1762491985.46 total 43.18 last 18.90) - open/close 912 (time 1762491995.66 total 53.37 last 11.67) - open/close 1056 (time 1762492006.19 total 63.91 last 13.67) - open/close 1279 (time 1762492017.08 total 74.79 last 20.48) - open/close 1439 (time 1762492027.97 total 85.68 last 14.70) - open/close 1614 (time 1762492038.77 total 96.49 last 16.19) - open/close 1788 (time 1762492049.28 total 106.99 last 16.56) - open/close 2000 (time 1762492059.29 total 117.01 last 21.16) - open/close 2166 (time 1762492069.30 total 127.02 last 16.58) - open/close 2403 (time 1762492080.11 total 137.83 last 21.92) - open/close 2632 (time 1762492090.94 total 148.65 last 21.16) - open/close 2892 (time 1762492100.95 total 158.66 last 25.97) - open/close 3100 (time 1762492111.97 total 169.68 last 18.88) - open/close 3288 (time 1762492122.54 total 180.26 last 17.77) - open/close 3427 (time 1762492132.79 total 190.51 last 13.56) - open/close 3540 (time 1762492142.80 total 200.51 last 11.29) - open/close 3768 (time 1762492152.80 total 210.51 last 22.79) - open/close 3937 (time 1762492162.80 total 220.52 last 16.89) - open/close 4117 (time 1762492172.81 total 230.52 last 17.99) - open/close 4326 (time 1762492182.84 total 240.56 last 20.83) - open/close 4499 (time 1762492193.53 total 251.25 last 16.18) - open/close 4701 (time 1762492204.40 total 262.12 last 18.59) - open/close 4927 (time 1762492214.41 total 272.12 last 22.58) - open/close 5127 (time 1762492224.43 total 282.15 last 19.96) - open/close 5353 (time 1762492234.82 total 292.53 last 21.76) - open/close 5550 (time 1762492244.83 total 302.54 last 19.68) - open/close 5725 (time 1762492254.83 total 312.55 last 17.49) - open/close 5890 (time 1762492264.91 total 322.63 last 16.37) - open/close 6093 (time 1762492275.86 total 333.57 last 18.55) - open/close 6300 (time 1762492285.86 total 343.58 last 20.68) - open/close 6507 (time 1762492295.98 total 353.70 last 20.46) - open/close 6750 (time 1762492306.77 total 364.49 last 22.52) - open/close 6950 (time 1762492316.79 total 374.51 last 19.97) - open/close 7122 (time 1762492327.54 total 385.26 last 16.00) - open/close 7306 (time 1762492337.55 total 395.26 last 18.39) - open/close 7493 (time 1762492348.22 total 405.94 last 17.52) - open/close 7700 (time 1762492358.25 total 415.96 last 20.65) - open/close 7918 (time 1762492368.64 total 426.36 last 20.97) - open/close 8140 (time 1762492379.13 total 436.85 last 21.16) - open/close 8387 (time 1762492389.51 total 447.23 last 23.79) - open/close 8528 (time 1762492400.13 total 457.84 last 13.29) - open/close 8723 (time 1762492410.62 total 468.33 last 18.59) - open/close 8900 (time 1762492420.63 total 478.34 last 17.68) - open/close 9064 (time 1762492430.63 total 488.35 last 16.40) - open/close 9233 (time 1762492440.63 total 498.35 last 16.90) - open/close 9405 (time 1762492451.39 total 509.11 last 15.98) - open/close 9653 (time 1762492462.10 total 519.82 last 23.16) - open/close 9849 (time 1762492472.13 total 529.84 last 19.55) - open/close 9912 (time 1762492484.98 total 542.70 last 4.90) - open/close 10000 (time 1762492489.94 total 547.65 last 17.74) - open/close 10226 (time 1762492500.49 total 558.20 last 21.43) - open/close 10419 (time 1762492510.49 total 568.21 last 19.28) - open/close 10654 (time 1762492520.50 total 578.22 last 23.48) - open/close 10831 (time 1762492530.93 total 588.64 last 16.98) - open/close 11037 (time 1762492540.94 total 598.66 last 20.57) - open/close 11273 (time 1762492550.97 total 608.68 last 23.54) - open/close 11452 (time 1762492560.98 total 618.69 last 17.88) - open/close 11641 (time 1762492571.38 total 629.09 last 18.17) - open/close 11828 (time 1762492581.40 total 639.12 last 18.65) - open/close 11973 (time 1762492591.74 total 649.46 last 14.03) - open/close 12138 (time 1762492602.24 total 659.96 last 15.72) - open/close 12296 (time 1762492612.24 total 669.96 last 15.80) - open/close 12457 (time 1762492623.17 total 680.89 last 14.73) - open/close 12663 (time 1762492633.20 total 690.92 last 20.55) - open/close 12869 (time 1762492644.15 total 701.87 last 18.81) - open/close 13099 (time 1762492654.16 total 711.88 last 22.97) - open/close 13259 (time 1762492664.16 total 721.88 last 16.00) - open/close 13437 (time 1762492674.29 total 732.01 last 17.58) - open/close 13610 (time 1762492684.80 total 742.52 last 16.46) - open/close 13815 (time 1762492695.36 total 753.07 last 19.42) - open/close 14015 (time 1762492705.85 total 763.57 last 19.06) - open/close 14201 (time 1762492716.72 total 774.43 last 17.11) - open/close 14495 (time 1762492726.72 total 784.44 last 29.39) - open/close 15336 (time 1762492736.73 total 794.44 last 84.08) - open/close 16095 (time 1762492746.74 total 804.45 last 75.82) - open/close 16798 (time 1762492756.74 total 814.45 last 70.30) - open/close 17528 (time 1762492766.74 total 824.46 last 72.96) - open/close 18246 (time 1762492776.75 total 834.47 last 71.72) - open/close 18940 (time 1762492786.77 total 844.48 last 69.28) - open/close 19699 (time 1762492796.77 total 854.49 last 75.88) - open/close 20000 (time 1762492800.69 total 858.41 last 76.76) - open/close 20707 (time 1762492810.70 total 868.42 last 70.64) - open/close 21421 (time 1762492820.70 total 878.42 last 71.39) - open/close 22053 (time 1762492830.71 total 888.42 last 63.17) - open/close 22689 (time 1762492840.71 total 898.43 last 63.57) - open/close 23182 (time 1762492850.72 total 908.43 last 49.28) - open/close 23838 (time 1762492860.72 total 918.44 last 65.56) - open/close 24526 (time 1762492870.73 total 928.45 last 68.74) - open/close 25233 (time 1762492880.74 total 938.46 last 70.63) - open/close 25986 (time 1762492890.75 total 948.47 last 75.24) - open/close 26753 (time 1762492900.76 total 958.47 last 76.63) - open/close 27529 (time 1762492910.76 total 968.47 last 77.60) - open/close 28235 (time 1762492920.76 total 978.47 last 70.60) - open/close 28914 (time 1762492930.77 total 988.48 last 67.84) - open/close 29631 (time 1762492940.78 total 998.50 last 71.60) - open/close 30000 (time 1762492945.89 total 1003.61 last 72.19) - open/close 30703 (time 1762492955.93 total 1013.65 last 70.04) - open/close 31412 (time 1762492965.94 total 1023.65 last 70.85) - open/close 32094 (time 1762492975.94 total 1033.66 last 68.16) - open/close 32767 (time 1762492985.95 total 1043.67 last 67.25) - open/close 33489 (time 1762492995.95 total 1053.67 last 72.18) - open/close 34299 (time 1762493005.97 total 1063.68 last 80.89) - open/close 34940 (time 1762493015.98 total 1073.70 last 64.01) - open/close 35507 (time 1762493025.99 total 1083.70 last 56.66) - open/close 36205 (time 1762493035.99 total 1093.71 last 69.78) - open/close 36800 (time 1762493046.00 total 1103.71 last 59.46) - open/close 37630 (time 1762493056.01 total 1113.73 last 82.89) - open/close 38307 (time 1762493066.02 total 1123.74 last 67.61) - open/close 39031 (time 1762493076.04 total 1133.76 last 72.28) - open/close 39716 (time 1762493086.05 total 1143.77 last 68.41) - open/close 40000 (time 1762493090.66 total 1148.37 last 61.71) - open/close 40624 (time 1762493100.66 total 1158.37 last 62.40) - open/close 41304 (time 1762493110.67 total 1168.38 last 67.93) - open/close 41963 (time 1762493120.68 total 1178.39 last 65.82) - open/close 42626 (time 1762493130.69 total 1188.40 last 66.24) - open/close 43276 (time 1762493140.69 total 1198.40 last 64.99) - open/close 43950 (time 1762493150.69 total 1208.41 last 67.38) - open/close 44627 (time 1762493160.70 total 1218.41 last 67.65) - open/close 45266 (time 1762493170.73 total 1228.44 last 63.69) - open/close 45895 (time 1762493180.74 total 1238.46 last 62.83) - open/close 46533 (time 1762493190.74 total 1248.46 last 63.79) - open/close 47159 (time 1762493200.75 total 1258.47 last 62.54) - open/close 47681 (time 1762493210.77 total 1268.48 last 52.13) - open/close 48316 (time 1762493220.78 total 1278.49 last 63.44) - open/close 48958 (time 1762493230.79 total 1288.50 last 64.14) - open/close 49537 (time 1762493240.80 total 1298.51 last 57.82) total: 50000 open/close in 1305.37 seconds: 38.30 ops/second - unlinked 0 (time 1762493251 ; total 0 ; last 0) - unlinked 10000 (time 1762493400 ; total 149 ; last 149) - unlinked 20000 (time 1762493542 ; total 291 ; last 142) - unlinked 30000 (time 1762493694 ; total 443 ; last 152) - unlinked 40000 (time 1762493844 ; total 593 ; last 150) total: 50000 unlinks in 743 seconds: 67.294754 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg622-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg622-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg622-server stop mds service on oleg622-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg622-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing unload_modules_local oleg622-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg622-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2