== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:46:31 (1758739591) start mds service on oleg224-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg224-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1758739662.41 total 10.92 last 20.25) - open/close 447 (time 1758739672.42 total 20.92 last 22.58) - open/close 676 (time 1758739682.44 total 30.94 last 22.85) - open/close 895 (time 1758739692.55 total 41.06 last 21.66) - open/close 1148 (time 1758739702.87 total 51.37 last 24.53) - open/close 1357 (time 1758739713.29 total 61.80 last 20.05) - open/close 1578 (time 1758739724.24 total 72.74 last 20.19) - open/close 1816 (time 1758739734.47 total 82.98 last 23.26) - open/close 2024 (time 1758739745.48 total 93.99 last 18.89) - open/close 2236 (time 1758739756.21 total 104.71 last 19.76) - open/close 2483 (time 1758739766.22 total 114.73 last 24.67) - open/close 2715 (time 1758739777.25 total 125.75 last 21.05) - open/close 2941 (time 1758739787.60 total 136.11 last 21.83) - open/close 3124 (time 1758739797.61 total 146.11 last 18.29) - open/close 3294 (time 1758739807.61 total 156.12 last 16.99) - open/close 3507 (time 1758739818.39 total 166.89 last 19.77) - open/close 3732 (time 1758739829.47 total 177.97 last 20.31) - open/close 3963 (time 1758739839.47 total 187.97 last 23.10) - open/close 4167 (time 1758739850.32 total 198.83 last 18.79) - open/close 4380 (time 1758739861.32 total 209.82 last 19.37) - open/close 4576 (time 1758739871.81 total 220.32 last 18.68) - open/close 4735 (time 1758739882.63 total 231.13 last 14.70) - open/close 4895 (time 1758739892.65 total 241.15 last 15.97) - open/close 5105 (time 1758739902.69 total 251.19 last 20.92) - open/close 5311 (time 1758739913.55 total 262.06 last 18.96) - open/close 5414 (time 1758739923.93 total 272.43 last 9.93) - open/close 5595 (time 1758739934.34 total 282.84 last 17.39) - open/close 5789 (time 1758739944.59 total 293.09 last 18.92) - open/close 5962 (time 1758739955.52 total 304.03 last 15.82) - open/close 6103 (time 1758739965.77 total 314.27 last 13.76) - open/close 6311 (time 1758739976.46 total 324.97 last 19.45) - open/close 6495 (time 1758739986.47 total 334.97 last 18.40) - open/close 6670 (time 1758739996.47 total 344.97 last 17.49) - open/close 6881 (time 1758740007.12 total 355.62 last 19.82) - open/close 7073 (time 1758740017.56 total 366.06 last 18.39) - open/close 7240 (time 1758740028.55 total 377.06 last 15.19) - open/close 7397 (time 1758740039.24 total 387.74 last 14.69) - open/close 7619 (time 1758740049.24 total 397.74 last 22.20) - open/close 7805 (time 1758740059.33 total 407.83 last 18.43) - open/close 8054 (time 1758740069.46 total 417.96 last 24.58) - open/close 8265 (time 1758740080.30 total 428.80 last 19.46) - open/close 8502 (time 1758740090.31 total 438.81 last 23.68) - open/close 8706 (time 1758740100.95 total 449.45 last 19.17) - open/close 8898 (time 1758740111.69 total 460.20 last 17.87) - open/close 9142 (time 1758740122.07 total 470.58 last 23.51) - open/close 9357 (time 1758740132.87 total 481.37 last 19.91) - open/close 9636 (time 1758740142.88 total 491.38 last 27.87) - open/close 9808 (time 1758740152.99 total 501.50 last 17.01) - open/close 9989 (time 1758740163.98 total 512.48 last 16.48) - open/close 10206 (time 1758740174.16 total 522.66 last 21.32) - open/close 10349 (time 1758740184.17 total 532.67 last 14.28) - open/close 10576 (time 1758740194.19 total 542.69 last 22.66) - open/close 10795 (time 1758740204.75 total 553.25 last 20.73) - open/close 10959 (time 1758740215.24 total 563.74 last 15.64) - open/close 11161 (time 1758740225.25 total 573.75 last 20.18) - open/close 11373 (time 1758740235.33 total 583.84 last 21.02) - open/close 11563 (time 1758740245.33 total 593.84 last 18.99) - open/close 11691 (time 1758740255.56 total 604.06 last 12.52) - open/close 11877 (time 1758740266.06 total 614.56 last 17.71) - open/close 12069 (time 1758740276.44 total 624.95 last 18.49) - open/close 12278 (time 1758740287.13 total 635.64 last 19.55) - open/close 12483 (time 1758740297.86 total 646.37 last 19.10) - open/close 12679 (time 1758740308.11 total 656.62 last 19.12) - open/close 12876 (time 1758740318.48 total 666.99 last 19.00) - open/close 13022 (time 1758740328.78 total 677.28 last 14.18) - open/close 13190 (time 1758740338.78 total 687.28 last 16.79) - open/close 13387 (time 1758740348.93 total 697.44 last 19.40) - open/close 13571 (time 1758740359.05 total 707.55 last 18.19) - open/close 13744 (time 1758740370.12 total 718.62 last 15.63) - open/close 13925 (time 1758740380.12 total 728.63 last 18.09) - open/close 14125 (time 1758740392.85 total 741.36 last 15.71) - open/close 14282 (time 1758740403.73 total 752.23 last 14.44) - open/close 14464 (time 1758740414.15 total 762.66 last 17.46) - open/close 15143 (time 1758740424.16 total 772.66 last 67.85) - open/close 15905 (time 1758740434.17 total 782.67 last 76.16) - open/close 16720 (time 1758740444.17 total 792.67 last 81.47) - open/close 17545 (time 1758740454.18 total 802.68 last 82.44) - open/close 18316 (time 1758740464.19 total 812.69 last 77.04) - open/close 19123 (time 1758740474.19 total 822.69 last 80.66) - open/close 19945 (time 1758740484.20 total 832.70 last 82.14) - open/close 20727 (time 1758740494.21 total 842.71 last 78.10) - open/close 21513 (time 1758740504.23 total 852.73 last 78.46) - open/close 22380 (time 1758740514.24 total 862.74 last 86.62) - open/close 23250 (time 1758740524.24 total 872.74 last 86.99) - open/close 24175 (time 1758740534.24 total 882.74 last 92.48) - open/close 25061 (time 1758740544.24 total 892.75 last 88.56) - open/close 25932 (time 1758740554.26 total 902.76 last 86.99) - open/close 26917 (time 1758740564.26 total 912.76 last 98.46) - open/close 27797 (time 1758740574.27 total 922.77 last 87.93) - open/close 28674 (time 1758740584.27 total 932.77 last 87.69) - open/close 29531 (time 1758740594.28 total 942.78 last 85.63) - open/close 30000 (time 1758740599.68 total 948.18 last 86.84) - open/close 30806 (time 1758740609.69 total 958.20 last 80.50) - open/close 31614 (time 1758740619.71 total 968.21 last 80.69) - open/close 32409 (time 1758740629.71 total 978.21 last 79.46) - open/close 33212 (time 1758740639.72 total 988.22 last 80.23) - open/close 34038 (time 1758740649.73 total 998.23 last 82.52) - open/close 35002 (time 1758740659.73 total 1008.23 last 96.37) - open/close 35879 (time 1758740669.74 total 1018.24 last 87.64) - open/close 36826 (time 1758740679.74 total 1028.25 last 94.65) - open/close 37716 (time 1758740689.76 total 1038.26 last 88.88) - open/close 38562 (time 1758740699.76 total 1048.26 last 84.58) - open/close 39391 (time 1758740709.76 total 1058.27 last 82.87) - open/close 40000 (time 1758740716.61 total 1065.11 last 89.01) - open/close 40919 (time 1758740726.62 total 1075.12 last 91.80) - open/close 41851 (time 1758740736.63 total 1085.13 last 93.10) - open/close 42782 (time 1758740746.63 total 1095.13 last 93.10) - open/close 43618 (time 1758740756.63 total 1105.14 last 83.56) - open/close 44476 (time 1758740766.64 total 1115.14 last 85.73) - open/close 45380 (time 1758740776.65 total 1125.15 last 90.34) - open/close 46343 (time 1758740786.70 total 1135.20 last 95.82) - open/close 47315 (time 1758740796.71 total 1145.21 last 97.09) - open/close 48220 (time 1758740806.71 total 1155.21 last 90.48) - open/close 49237 (time 1758740816.71 total 1165.22 last 101.68) total: 50000 open/close in 1172.07 seconds: 42.66 ops/second - unlinked 0 (time 1758740826 ; total 0 ; last 0) - unlinked 10000 (time 1758740925 ; total 99 ; last 99) - unlinked 20000 (time 1758740987 ; total 161 ; last 62) - unlinked 30000 (time 1758741029 ; total 203 ; last 42) - unlinked 40000 (time 1758741070 ; total 244 ; last 41) total: 50000 unlinks in 281 seconds: 177.935944 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg224-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2