== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:13:22 (1763082802) start mds service on oleg332-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 219 (time 1763082872.04 total 10.01 last 21.87) - open/close 435 (time 1763082882.04 total 20.02 last 21.59) - open/close 643 (time 1763082892.05 total 30.03 last 20.79) - open/close 860 (time 1763082903.03 total 41.01 last 19.76) - open/close 1069 (time 1763082913.51 total 51.49 last 19.93) - open/close 1285 (time 1763082924.51 total 62.49 last 19.64) - open/close 1471 (time 1763082934.70 total 72.68 last 18.26) - open/close 1632 (time 1763082944.70 total 82.68 last 16.09) - open/close 1803 (time 1763082956.65 total 94.63 last 14.31) - open/close 1994 (time 1763082966.95 total 104.92 last 18.55) - open/close 2161 (time 1763082977.31 total 115.29 last 16.11) - open/close 2344 (time 1763082987.63 total 125.61 last 17.74) - open/close 2529 (time 1763082997.88 total 135.85 last 18.05) - open/close 2755 (time 1763083007.89 total 145.87 last 22.57) - open/close 3007 (time 1763083018.79 total 156.77 last 23.12) - open/close 3216 (time 1763083029.82 total 167.80 last 18.95) - open/close 3447 (time 1763083040.34 total 178.31 last 21.96) - open/close 3710 (time 1763083050.99 total 188.96 last 24.69) - open/close 3888 (time 1763083061.60 total 199.58 last 16.76) - open/close 4176 (time 1763083071.85 total 209.83 last 28.10) - open/close 4389 (time 1763083082.06 total 220.04 last 20.87) - open/close 4577 (time 1763083092.80 total 230.78 last 17.50) - open/close 4768 (time 1763083102.83 total 240.81 last 19.05) - open/close 5024 (time 1763083112.84 total 250.81 last 25.58) - open/close 5217 (time 1763083123.17 total 261.15 last 18.67) - open/close 5429 (time 1763083134.13 total 272.11 last 19.35) - open/close 5636 (time 1763083145.02 total 283.00 last 19.00) - open/close 5841 (time 1763083156.01 total 293.98 last 18.66) - open/close 6059 (time 1763083166.96 total 304.94 last 19.90) - open/close 6286 (time 1763083176.97 total 314.94 last 22.69) - open/close 6513 (time 1763083186.98 total 324.96 last 22.67) - open/close 6732 (time 1763083198.25 total 336.23 last 19.43) - open/close 6924 (time 1763083208.27 total 346.25 last 19.17) - open/close 7129 (time 1763083218.28 total 356.26 last 20.48) - open/close 7334 (time 1763083228.30 total 366.28 last 20.47) - open/close 7548 (time 1763083238.38 total 376.35 last 21.23) - open/close 7849 (time 1763083249.08 total 387.06 last 28.12) - open/close 8065 (time 1763083259.08 total 397.06 last 21.59) - open/close 8295 (time 1763083269.09 total 407.07 last 22.98) - open/close 8508 (time 1763083279.10 total 417.08 last 21.27) - open/close 8714 (time 1763083289.84 total 427.81 last 19.20) - open/close 8905 (time 1763083300.32 total 438.30 last 18.22) - open/close 9085 (time 1763083311.34 total 449.32 last 16.32) - open/close 9264 (time 1763083321.35 total 459.33 last 17.89) - open/close 9449 (time 1763083331.63 total 469.60 last 18.00) - open/close 9630 (time 1763083342.00 total 479.98 last 17.44) - open/close 9830 (time 1763083352.55 total 490.53 last 18.96) - open/close 10000 (time 1763083359.72 total 497.70 last 23.69) - open/close 10253 (time 1763083369.74 total 507.71 last 25.27) - open/close 10473 (time 1763083379.75 total 517.73 last 21.97) - open/close 10621 (time 1763083390.63 total 528.61 last 13.60) - open/close 10788 (time 1763083401.65 total 539.63 last 15.15) - open/close 11022 (time 1763083412.59 total 550.56 last 21.40) - open/close 11183 (time 1763083423.16 total 561.14 last 15.22) - open/close 11327 (time 1763083433.70 total 571.68 last 13.67) - open/close 11502 (time 1763083443.72 total 581.69 last 17.47) - open/close 11679 (time 1763083454.63 total 592.61 last 16.21) - open/close 11799 (time 1763083465.52 total 603.50 last 11.03) - open/close 11929 (time 1763083475.64 total 613.62 last 12.84) - open/close 12092 (time 1763083486.30 total 624.28 last 15.29) - open/close 12242 (time 1763083496.68 total 634.66 last 14.45) - open/close 12407 (time 1763083507.30 total 645.28 last 15.54) - open/close 12518 (time 1763083517.30 total 655.28 last 11.10) - open/close 12685 (time 1763083528.11 total 666.09 last 15.45) - open/close 12858 (time 1763083539.06 total 677.04 last 15.80) - open/close 13049 (time 1763083549.55 total 687.53 last 18.21) - open/close 13281 (time 1763083560.56 total 698.54 last 21.06) - open/close 13422 (time 1763083570.61 total 708.59 last 14.04) - open/close 13517 (time 1763083581.23 total 719.21 last 8.94) - open/close 13641 (time 1763083591.98 total 729.96 last 11.54) - open/close 14092 (time 1763083601.99 total 739.97 last 45.08) - open/close 15068 (time 1763083611.99 total 749.97 last 97.58) - open/close 16338 (time 1763083621.99 total 759.97 last 126.94) - open/close 17471 (time 1763083632.00 total 769.98 last 113.26) - open/close 18541 (time 1763083642.00 total 779.98 last 106.93) - open/close 19600 (time 1763083652.01 total 789.99 last 105.84) - open/close 20000 (time 1763083655.47 total 793.45 last 115.64) - open/close 20996 (time 1763083665.47 total 803.45 last 99.59) - open/close 21863 (time 1763083675.47 total 813.45 last 86.66) - open/close 22763 (time 1763083685.48 total 823.46 last 89.91) - open/close 23719 (time 1763083695.49 total 833.47 last 95.52) - open/close 24651 (time 1763083705.50 total 843.47 last 93.16) - open/close 25699 (time 1763083715.50 total 853.48 last 104.79) - open/close 26728 (time 1763083725.50 total 863.48 last 102.85) - open/close 27825 (time 1763083735.52 total 873.50 last 109.53) - open/close 28919 (time 1763083745.52 total 883.50 last 109.37) - open/close 29901 (time 1763083755.52 total 893.50 last 98.16) - open/close 30864 (time 1763083765.53 total 903.51 last 96.26) - open/close 31872 (time 1763083775.53 total 913.51 last 100.76) - open/close 33148 (time 1763083785.56 total 923.53 last 127.31) - open/close 34089 (time 1763083795.57 total 933.55 last 93.98) - open/close 35156 (time 1763083805.57 total 943.55 last 106.67) - open/close 36391 (time 1763083815.58 total 953.55 last 123.44) - open/close 37803 (time 1763083825.58 total 963.56 last 141.13) - open/close 39024 (time 1763083835.59 total 973.57 last 121.95) - open/close 40000 (time 1763083844.74 total 982.71 last 106.77) - open/close 41102 (time 1763083854.74 total 992.71 last 110.19) - open/close 42515 (time 1763083864.74 total 1002.72 last 141.27) - open/close 43793 (time 1763083874.74 total 1012.72 last 127.74) - open/close 45123 (time 1763083884.74 total 1022.72 last 132.98) - open/close 46627 (time 1763083894.75 total 1032.73 last 150.36) - open/close 47951 (time 1763083904.75 total 1042.73 last 132.35) - open/close 49245 (time 1763083914.76 total 1052.74 last 129.27) total: 50000 open/close in 1058.74 seconds: 47.23 ops/second - unlinked 0 (time 1763083922 ; total 0 ; last 0) - unlinked 10000 (time 1763084022 ; total 100 ; last 100) - unlinked 20000 (time 1763084070 ; total 148 ; last 48) - unlinked 30000 (time 1763084112 ; total 190 ; last 42) - unlinked 40000 (time 1763084141 ; total 219 ; last 29) total: 50000 unlinks in 250 seconds: 200.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg332-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2