== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:33:53 (1763156033) start mds service on oleg439-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg439-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1763156108.56 total 10.93 last 20.95) - open/close 448 (time 1763156118.58 total 20.95 last 21.86) - open/close 644 (time 1763156128.60 total 30.97 last 19.57) - open/close 854 (time 1763156139.48 total 41.84 last 19.30) - open/close 1006 (time 1763156150.17 total 52.54 last 14.22) - open/close 1177 (time 1763156161.16 total 63.53 last 15.55) - open/close 1311 (time 1763156171.66 total 74.02 last 12.77) - open/close 1454 (time 1763156182.17 total 84.54 last 13.60) - open/close 1593 (time 1763156192.78 total 95.15 last 13.10) - open/close 1764 (time 1763156203.61 total 105.98 last 15.79) - open/close 1900 (time 1763156213.97 total 116.34 last 13.13) - open/close 2109 (time 1763156223.99 total 126.36 last 20.86) - open/close 2323 (time 1763156234.00 total 136.37 last 21.38) - open/close 2483 (time 1763156245.08 total 147.45 last 14.43) - open/close 2652 (time 1763156256.08 total 158.44 last 15.37) - open/close 2867 (time 1763156266.97 total 169.34 last 19.74) - open/close 3098 (time 1763156277.97 total 180.34 last 20.99) - open/close 3235 (time 1763156288.33 total 190.70 last 13.23) - open/close 3358 (time 1763156298.51 total 200.88 last 12.08) - open/close 3477 (time 1763156308.52 total 210.89 last 11.89) - open/close 3675 (time 1763156318.93 total 221.30 last 19.02) - open/close 3807 (time 1763156329.10 total 231.47 last 12.98) - open/close 3923 (time 1763156339.14 total 241.51 last 11.55) - open/close 4028 (time 1763156349.99 total 252.35 last 9.68) - open/close 4201 (time 1763156360.00 total 262.37 last 17.27) - open/close 4344 (time 1763156370.52 total 272.88 last 13.60) - open/close 4463 (time 1763156381.39 total 283.75 last 10.95) - open/close 4585 (time 1763156391.40 total 293.77 last 12.19) - open/close 4738 (time 1763156402.51 total 304.87 last 13.77) - open/close 4876 (time 1763156412.75 total 315.12 last 13.47) - open/close 5065 (time 1763156423.38 total 325.75 last 17.78) - open/close 5192 (time 1763156433.55 total 335.92 last 12.48) - open/close 5447 (time 1763156444.36 total 346.73 last 23.59) - open/close 5542 (time 1763156454.80 total 357.17 last 9.10) - open/close 5641 (time 1763156465.56 total 367.93 last 9.20) - open/close 5805 (time 1763156475.57 total 377.94 last 16.39) - open/close 5986 (time 1763156485.57 total 387.94 last 18.09) - open/close 6168 (time 1763156495.63 total 398.00 last 18.09) - open/close 6335 (time 1763156505.68 total 408.04 last 16.63) - open/close 6437 (time 1763156516.45 total 418.82 last 9.47) - open/close 6495 (time 1763156526.55 total 428.91 last 5.74) - open/close 6641 (time 1763156536.97 total 439.34 last 14.01) - open/close 6818 (time 1763156548.19 total 450.56 last 15.78) - open/close 6869 (time 1763156559.19 total 461.55 last 4.64) - open/close 6973 (time 1763156569.87 total 472.24 last 9.73) - open/close 7086 (time 1763156580.70 total 483.06 last 10.44) - open/close 7228 (time 1763156590.95 total 493.32 last 13.85) - open/close 7424 (time 1763156601.10 total 503.47 last 19.32) - open/close 7640 (time 1763156611.42 total 513.79 last 20.93) - open/close 7939 (time 1763156621.42 total 523.79 last 29.90) - open/close 8158 (time 1763156632.27 total 534.64 last 20.19) - open/close 8348 (time 1763156642.63 total 545.00 last 18.33) - open/close 8557 (time 1763156653.60 total 555.97 last 19.05) - open/close 8735 (time 1763156663.63 total 566.00 last 17.74) - open/close 8985 (time 1763156674.13 total 576.50 last 23.81) - open/close 9188 (time 1763156684.95 total 587.32 last 18.76) - open/close 9371 (time 1763156694.96 total 597.33 last 18.29) - open/close 9573 (time 1763156705.43 total 607.80 last 19.29) - open/close 9752 (time 1763156715.66 total 618.03 last 17.50) - open/close 9948 (time 1763156726.54 total 628.91 last 18.02) - open/close 10000 (time 1763156729.40 total 631.77 last 18.17) - open/close 10123 (time 1763156739.47 total 641.84 last 12.21) - open/close 10291 (time 1763156749.64 total 652.01 last 16.52) - open/close 10479 (time 1763156759.66 total 662.03 last 18.77) - open/close 11252 (time 1763156769.67 total 672.04 last 77.23) - open/close 12210 (time 1763156779.68 total 682.05 last 95.72) - open/close 13109 (time 1763156789.68 total 692.05 last 89.90) - open/close 14049 (time 1763156799.68 total 702.05 last 93.96) - open/close 14972 (time 1763156809.69 total 712.06 last 92.24) - open/close 15851 (time 1763156819.69 total 722.06 last 87.87) - open/close 16591 (time 1763156829.70 total 732.06 last 73.98) - open/close 17412 (time 1763156839.70 total 742.07 last 82.03) - open/close 18230 (time 1763156849.71 total 752.07 last 81.78) - open/close 18983 (time 1763156859.71 total 762.08 last 75.24) - open/close 19785 (time 1763156869.72 total 772.09 last 80.13) - open/close 20000 (time 1763156872.63 total 775.00 last 73.98) - open/close 20866 (time 1763156882.63 total 785.00 last 86.55) - open/close 21626 (time 1763156892.64 total 795.01 last 75.94) - open/close 22353 (time 1763156902.65 total 805.01 last 72.68) - open/close 23086 (time 1763156912.65 total 815.02 last 73.25) - open/close 23890 (time 1763156922.66 total 825.02 last 80.37) - open/close 24755 (time 1763156932.66 total 835.03 last 86.43) - open/close 25680 (time 1763156942.67 total 845.04 last 92.47) - open/close 26744 (time 1763156952.68 total 855.05 last 106.24) - open/close 27651 (time 1763156962.69 total 865.05 last 90.68) - open/close 28487 (time 1763156972.69 total 875.06 last 83.55) - open/close 29326 (time 1763156982.70 total 885.07 last 83.85) - open/close 30000 (time 1763156989.78 total 892.15 last 95.15) - open/close 30848 (time 1763156999.79 total 902.16 last 84.68) - open/close 31714 (time 1763157009.80 total 912.17 last 86.54) - open/close 32651 (time 1763157019.81 total 922.18 last 93.63) - open/close 33634 (time 1763157029.81 total 932.18 last 98.27) - open/close 34631 (time 1763157039.83 total 942.20 last 99.55) - open/close 35476 (time 1763157049.83 total 952.20 last 84.46) - open/close 36428 (time 1763157059.84 total 962.21 last 95.15) - open/close 37337 (time 1763157069.84 total 972.21 last 90.85) - open/close 38330 (time 1763157079.85 total 982.22 last 99.27) - open/close 39368 (time 1763157089.85 total 992.22 last 103.77) - open/close 40000 (time 1763157095.92 total 998.29 last 104.10) - open/close 40921 (time 1763157105.93 total 1008.30 last 92.00) - open/close 41808 (time 1763157115.94 total 1018.30 last 88.67) - open/close 42728 (time 1763157125.94 total 1028.31 last 91.94) - open/close 43733 (time 1763157135.95 total 1038.31 last 100.46) - open/close 44613 (time 1763157145.95 total 1048.32 last 87.94) - open/close 45464 (time 1763157155.95 total 1058.32 last 85.08) - open/close 46394 (time 1763157165.96 total 1068.33 last 92.91) - open/close 47492 (time 1763157175.97 total 1078.34 last 109.74) - open/close 48506 (time 1763157185.98 total 1088.35 last 101.31) - open/close 49440 (time 1763157195.98 total 1098.35 last 93.36) total: 50000 open/close in 1103.41 seconds: 45.31 ops/second - unlinked 0 (time 1763157203 ; total 0 ; last 0) - unlinked 10000 (time 1763157290 ; total 87 ; last 87) - unlinked 20000 (time 1763157334 ; total 131 ; last 44) - unlinked 30000 (time 1763157382 ; total 179 ; last 48) - unlinked 40000 (time 1763157425 ; total 222 ; last 43) total: 50000 unlinks in 265 seconds: 188.679245 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local oleg439-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg439-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2