== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:19:01 (1764735541) start mds service on oleg303-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 70 (time 1764735606.81 total 10.47 last 6.69) - open/close 220 (time 1764735617.74 total 21.40 last 13.72) - open/close 490 (time 1764735628.29 total 31.95 last 25.58) - open/close 765 (time 1764735639.33 total 42.98 last 24.93) - open/close 905 (time 1764735649.36 total 53.02 last 13.95) - open/close 1078 (time 1764735660.81 total 64.47 last 15.11) - open/close 1275 (time 1764735671.18 total 74.83 last 19.01) - open/close 1526 (time 1764735682.06 total 85.72 last 23.06) - open/close 1627 (time 1764735692.69 total 96.35 last 9.50) - open/close 1855 (time 1764735702.70 total 106.36 last 22.79) - open/close 2082 (time 1764735713.36 total 117.02 last 21.30) - open/close 2355 (time 1764735723.41 total 127.07 last 27.15) - open/close 2649 (time 1764735733.42 total 137.08 last 29.39) - open/close 2790 (time 1764735743.69 total 147.35 last 13.72) - open/close 3028 (time 1764735753.70 total 157.36 last 23.78) - open/close 3189 (time 1764735763.71 total 167.36 last 16.10) - open/close 3456 (time 1764735774.41 total 178.07 last 24.94) - open/close 3760 (time 1764735784.78 total 188.44 last 29.31) - open/close 4014 (time 1764735795.40 total 199.06 last 23.92) - open/close 4250 (time 1764735805.77 total 209.43 last 22.77) - open/close 4531 (time 1764735818.83 total 222.49 last 21.51) - open/close 4814 (time 1764735828.83 total 232.49 last 28.30) - open/close 5083 (time 1764735839.63 total 243.29 last 24.90) - open/close 5377 (time 1764735850.51 total 254.16 last 27.04) - open/close 5661 (time 1764735860.56 total 264.22 last 28.25) - open/close 6002 (time 1764735870.92 total 274.58 last 32.91) - open/close 6196 (time 1764735881.03 total 284.69 last 19.18) - open/close 6403 (time 1764735891.53 total 295.19 last 19.72) - open/close 6712 (time 1764735902.28 total 305.94 last 28.74) - open/close 7019 (time 1764735913.22 total 316.88 last 28.06) - open/close 7289 (time 1764735923.79 total 327.45 last 25.55) - open/close 7596 (time 1764735934.15 total 337.81 last 29.62) - open/close 7824 (time 1764735944.97 total 348.63 last 21.08) - open/close 8100 (time 1764735955.73 total 359.39 last 25.65) - open/close 8410 (time 1764735966.47 total 370.13 last 28.85) - open/close 8643 (time 1764735976.48 total 380.14 last 23.28) - open/close 8866 (time 1764735986.48 total 390.14 last 22.29) - open/close 9130 (time 1764735998.79 total 402.45 last 21.45) - open/close 9388 (time 1764736009.24 total 412.89 last 24.71) - open/close 9651 (time 1764736019.27 total 422.93 last 26.21) - open/close 9879 (time 1764736029.96 total 433.62 last 21.32) - open/close 10000 (time 1764736033.02 total 436.68 last 39.60) - open/close 10308 (time 1764736043.92 total 447.58 last 28.26) - open/close 10554 (time 1764736053.92 total 457.58 last 24.60) - open/close 11063 (time 1764736063.92 total 467.58 last 50.90) - open/close 12993 (time 1764736073.92 total 477.58 last 192.95) - open/close 14948 (time 1764736083.93 total 487.59 last 195.39) - open/close 16474 (time 1764736093.93 total 497.59 last 152.54) - open/close 17968 (time 1764736103.94 total 507.59 last 149.35) - open/close 19362 (time 1764736113.94 total 517.60 last 139.29) - open/close 20000 (time 1764736118.51 total 522.17 last 139.76) - open/close 21361 (time 1764736128.53 total 532.18 last 135.85) - open/close 22948 (time 1764736138.53 total 542.19 last 158.60) - open/close 24507 (time 1764736148.54 total 552.20 last 155.81) - open/close 25527 (time 1764736158.54 total 562.20 last 101.99) - open/close 26151 (time 1764736168.56 total 572.22 last 62.27) - open/close 26716 (time 1764736178.57 total 582.22 last 56.48) - open/close 27330 (time 1764736188.58 total 592.24 last 61.30) - open/close 27889 (time 1764736198.59 total 602.25 last 55.85) - open/close 28478 (time 1764736208.60 total 612.26 last 58.83) - open/close 29104 (time 1764736218.61 total 622.26 last 62.58) - open/close 29838 (time 1764736228.61 total 632.26 last 73.39) - open/close 30624 (time 1764736238.63 total 642.29 last 78.40) - open/close 31416 (time 1764736248.64 total 652.29 last 79.17) - open/close 32263 (time 1764736258.64 total 662.29 last 84.69) - open/close 33089 (time 1764736268.65 total 672.30 last 82.53) - open/close 33844 (time 1764736278.65 total 682.31 last 75.48) - open/close 34633 (time 1764736288.65 total 692.31 last 78.89) - open/close 35386 (time 1764736298.66 total 702.31 last 75.26) - open/close 36124 (time 1764736308.66 total 712.32 last 73.76) - open/close 36900 (time 1764736318.67 total 722.32 last 77.55) - open/close 37731 (time 1764736328.68 total 732.34 last 82.96) - open/close 38608 (time 1764736338.69 total 742.35 last 87.65) - open/close 39608 (time 1764736348.71 total 752.37 last 99.81) - open/close 40000 (time 1764736350.89 total 754.54 last 179.90) - open/close 41072 (time 1764736360.90 total 764.56 last 107.08) - open/close 41917 (time 1764736370.91 total 774.57 last 84.38) - open/close 42941 (time 1764736380.93 total 784.59 last 102.23) - open/close 43887 (time 1764736390.94 total 794.59 last 94.53) - open/close 44790 (time 1764736400.95 total 804.61 last 90.20) - open/close 45586 (time 1764736410.96 total 814.62 last 79.53) - open/close 46425 (time 1764736420.96 total 824.62 last 83.89) - open/close 47189 (time 1764736430.96 total 834.62 last 76.36) - open/close 47946 (time 1764736440.98 total 844.64 last 75.59) - open/close 48683 (time 1764736450.98 total 854.64 last 73.69) - open/close 49451 (time 1764736460.98 total 864.64 last 76.75) total: 50000 open/close in 871.26 seconds: 57.39 ops/second - unlinked 0 (time 1764736470 ; total 0 ; last 0) - unlinked 10000 (time 1764736584 ; total 114 ; last 114) - unlinked 20000 (time 1764736640 ; total 170 ; last 56) - unlinked 30000 (time 1764736693 ; total 223 ; last 53) - unlinked 40000 (time 1764736756 ; total 286 ; last 63) total: 50000 unlinks in 355 seconds: 140.845078 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg303-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2