== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:56:27 (1763351787) start mds service on oleg340-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg340-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg340-server: detected 4 online CPUs by sysfs oleg340-server: MODOPTS_LIBCFS= oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 173 (time 1763351851.07 total 10.01 last 17.28) - open/close 313 (time 1763351861.21 total 20.16 last 13.80) - open/close 505 (time 1763351871.27 total 30.21 last 19.10) - open/close 655 (time 1763351881.58 total 40.52 last 14.55) - open/close 778 (time 1763351891.58 total 50.52 last 12.30) - open/close 934 (time 1763351901.99 total 60.94 last 14.97) - open/close 1110 (time 1763351912.94 total 71.88 last 16.08) - open/close 1270 (time 1763351922.94 total 81.89 last 16.00) - open/close 1420 (time 1763351932.95 total 91.89 last 14.99) - open/close 1567 (time 1763351942.96 total 101.90 last 14.68) - open/close 1701 (time 1763351952.97 total 111.91 last 13.38) - open/close 1837 (time 1763351963.05 total 121.99 last 13.50) - open/close 2017 (time 1763351973.54 total 132.48 last 17.16) - open/close 2135 (time 1763351983.55 total 142.49 last 11.79) - open/close 2284 (time 1763351993.89 total 152.83 last 14.41) - open/close 2435 (time 1763352004.08 total 163.02 last 14.82) - open/close 2623 (time 1763352014.08 total 173.02 last 18.80) - open/close 2736 (time 1763352024.88 total 183.82 last 10.47) - open/close 2843 (time 1763352035.81 total 194.75 last 9.78) - open/close 3003 (time 1763352046.06 total 205.00 last 15.62) - open/close 3120 (time 1763352056.90 total 215.84 last 10.79) - open/close 3259 (time 1763352066.92 total 225.86 last 13.87) - open/close 3462 (time 1763352077.73 total 236.67 last 18.77) - open/close 3728 (time 1763352088.55 total 247.49 last 24.59) - open/close 3858 (time 1763352099.30 total 258.24 last 12.09) - open/close 4047 (time 1763352109.30 total 268.25 last 18.89) - open/close 4194 (time 1763352120.36 total 279.30 last 13.30) - open/close 4314 (time 1763352131.31 total 290.26 last 10.96) - open/close 4453 (time 1763352141.66 total 300.60 last 13.43) - open/close 4561 (time 1763352152.35 total 311.30 last 10.10) - open/close 4777 (time 1763352162.98 total 321.92 last 20.33) - open/close 4921 (time 1763352174.82 total 333.76 last 12.16) - open/close 5066 (time 1763352185.83 total 344.77 last 13.17) - open/close 5196 (time 1763352196.64 total 355.58 last 12.03) - open/close 5253 (time 1763352206.76 total 365.70 last 5.63) - open/close 5457 (time 1763352216.77 total 375.72 last 20.37) - open/close 5594 (time 1763352229.80 total 388.74 last 10.52) - open/close 5709 (time 1763352239.81 total 398.75 last 11.49) - open/close 5849 (time 1763352249.90 total 408.84 last 13.87) - open/close 5999 (time 1763352259.94 total 418.88 last 14.94) - open/close 6090 (time 1763352270.50 total 429.44 last 8.62) - open/close 6208 (time 1763352280.62 total 439.56 last 11.66) - open/close 6379 (time 1763352291.30 total 450.24 last 16.01) - open/close 6526 (time 1763352302.25 total 461.19 last 13.43) - open/close 6686 (time 1763352312.42 total 471.36 last 15.73) - open/close 6832 (time 1763352322.48 total 481.42 last 14.51) - open/close 6997 (time 1763352332.53 total 491.47 last 16.43) - open/close 7211 (time 1763352343.46 total 502.41 last 19.57) - open/close 7305 (time 1763352354.09 total 513.03 last 8.85) - open/close 7441 (time 1763352364.10 total 523.04 last 13.58) - open/close 7615 (time 1763352375.14 total 534.08 last 15.76) - open/close 7726 (time 1763352385.27 total 544.21 last 10.96) - open/close 7963 (time 1763352396.08 total 555.02 last 21.91) - open/close 8161 (time 1763352407.60 total 566.54 last 17.19) - open/close 8243 (time 1763352417.83 total 576.78 last 8.01) - open/close 8357 (time 1763352428.79 total 587.73 last 10.40) - open/close 9135 (time 1763352438.80 total 597.74 last 77.78) - open/close 10000 (time 1763352448.54 total 607.49 last 88.73) - open/close 10948 (time 1763352458.55 total 617.49 last 94.73) - open/close 11904 (time 1763352468.56 total 627.50 last 95.53) - open/close 12854 (time 1763352478.56 total 637.50 last 94.98) - open/close 13892 (time 1763352488.56 total 647.50 last 103.79) - open/close 14858 (time 1763352498.57 total 657.51 last 96.55) - open/close 15767 (time 1763352508.57 total 667.51 last 90.86) - open/close 16702 (time 1763352518.57 total 677.52 last 93.46) - open/close 17624 (time 1763352528.58 total 687.52 last 92.14) - open/close 18564 (time 1763352538.59 total 697.53 last 93.91) - open/close 19470 (time 1763352548.59 total 707.53 last 90.58) - open/close 20000 (time 1763352554.56 total 713.50 last 88.88) - open/close 20831 (time 1763352564.56 total 723.50 last 83.06) - open/close 21680 (time 1763352574.56 total 733.50 last 84.89) - open/close 22558 (time 1763352584.57 total 743.51 last 87.73) - open/close 23441 (time 1763352594.58 total 753.52 last 88.24) - open/close 24298 (time 1763352604.58 total 763.52 last 85.65) - open/close 25183 (time 1763352614.59 total 773.53 last 88.47) - open/close 26000 (time 1763352624.59 total 783.53 last 81.67) - open/close 26851 (time 1763352634.60 total 793.54 last 85.04) - open/close 27713 (time 1763352644.60 total 803.54 last 86.14) - open/close 28571 (time 1763352654.61 total 813.55 last 85.74) - open/close 29377 (time 1763352664.62 total 823.56 last 80.55) - open/close 30000 (time 1763352672.20 total 831.14 last 82.18) - open/close 30930 (time 1763352682.20 total 841.14 last 92.96) - open/close 31870 (time 1763352692.20 total 851.14 last 93.98) - open/close 32858 (time 1763352702.20 total 861.15 last 98.79) - open/close 33810 (time 1763352712.20 total 871.15 last 95.19) - open/close 35052 (time 1763352722.21 total 881.15 last 124.12) - open/close 35987 (time 1763352732.22 total 891.17 last 93.38) - open/close 36884 (time 1763352742.23 total 901.17 last 89.69) - open/close 37982 (time 1763352752.24 total 911.18 last 109.69) - open/close 38954 (time 1763352762.24 total 921.18 last 97.13) - open/close 39918 (time 1763352772.24 total 931.19 last 96.38) - open/close 40899 (time 1763352782.26 total 941.20 last 97.97) - open/close 41907 (time 1763352792.27 total 951.21 last 100.68) - open/close 42952 (time 1763352802.27 total 961.21 last 104.48) - open/close 43833 (time 1763352812.28 total 971.22 last 88.08) - open/close 44852 (time 1763352822.28 total 981.22 last 101.89) - open/close 46035 (time 1763352832.28 total 991.22 last 118.24) - open/close 47102 (time 1763352842.28 total 1001.22 last 106.69) - open/close 48154 (time 1763352852.28 total 1011.23 last 105.17) - open/close 49212 (time 1763352862.29 total 1021.23 last 105.78) total: 50000 open/close in 1028.88 seconds: 48.60 ops/second - unlinked 0 (time 1763352872 ; total 0 ; last 0) - unlinked 10000 (time 1763352935 ; total 63 ; last 63) - unlinked 20000 (time 1763352975 ; total 103 ; last 40) - unlinked 30000 (time 1763353012 ; total 140 ; last 37) - unlinked 40000 (time 1763353047 ; total 175 ; last 35) total: 50000 unlinks in 204 seconds: 245.098038 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg340-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2