== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:53:25 (1777499605) start mds service on oleg355-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg355-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1777499657.32 total 10.20 last 12.85) - open/close 279 (time 1777499667.63 total 20.50 last 14.36) - open/close 434 (time 1777499677.93 total 30.81 last 15.04) - open/close 660 (time 1777499688.50 total 41.38 last 21.38) - open/close 873 (time 1777499698.73 total 51.61 last 20.82) - open/close 999 (time 1777499708.75 total 61.62 last 12.58) - open/close 1197 (time 1777499719.79 total 72.67 last 17.92) - open/close 1290 (time 1777499730.41 total 83.29 last 8.76) - open/close 1432 (time 1777499740.59 total 93.46 last 13.96) - open/close 1618 (time 1777499751.22 total 104.09 last 17.50) - open/close 1797 (time 1777499761.77 total 114.65 last 16.96) - open/close 2006 (time 1777499772.59 total 125.47 last 19.31) - open/close 2122 (time 1777499782.60 total 135.48 last 11.59) - open/close 2299 (time 1777499792.61 total 145.49 last 17.68) - open/close 2476 (time 1777499803.64 total 156.52 last 16.04) - open/close 2689 (time 1777499814.46 total 167.34 last 19.69) - open/close 2874 (time 1777499824.59 total 177.47 last 18.26) - open/close 3078 (time 1777499834.93 total 187.80 last 19.74) - open/close 3268 (time 1777499845.94 total 198.81 last 17.26) - open/close 3440 (time 1777499856.69 total 209.57 last 15.99) - open/close 3631 (time 1777499866.93 total 219.80 last 18.66) - open/close 3857 (time 1777499877.94 total 230.81 last 20.53) - open/close 4004 (time 1777499888.30 total 241.18 last 14.19) - open/close 4181 (time 1777499898.30 total 251.18 last 17.70) - open/close 4355 (time 1777499909.10 total 261.97 last 16.12) - open/close 4496 (time 1777499919.35 total 272.22 last 13.76) - open/close 4652 (time 1777499929.98 total 282.85 last 14.67) - open/close 4794 (time 1777499942.90 total 295.78 last 10.99) - open/close 5045 (time 1777499953.52 total 306.40 last 23.64) - open/close 5256 (time 1777499963.96 total 316.84 last 20.21) - open/close 5385 (time 1777499973.97 total 326.84 last 12.89) - open/close 5529 (time 1777499984.36 total 337.24 last 13.85) - open/close 5651 (time 1777499995.31 total 348.19 last 11.14) - open/close 5779 (time 1777500005.32 total 358.20 last 12.79) - open/close 5899 (time 1777500015.33 total 368.20 last 11.99) - open/close 6062 (time 1777500025.66 total 378.53 last 15.78) - open/close 6323 (time 1777500036.20 total 389.07 last 24.76) - open/close 6645 (time 1777500046.25 total 399.12 last 32.04) - open/close 6966 (time 1777500057.13 total 410.01 last 29.49) - open/close 7212 (time 1777500067.88 total 420.76 last 22.88) - open/close 7472 (time 1777500078.38 total 431.26 last 24.77) - open/close 7713 (time 1777500089.00 total 441.88 last 22.69) - open/close 7994 (time 1777500099.01 total 451.88 last 28.09) - open/close 8211 (time 1777500109.35 total 462.23 last 20.97) - open/close 8328 (time 1777500119.92 total 472.79 last 11.08) - open/close 8684 (time 1777500130.99 total 483.86 last 32.16) - open/close 8995 (time 1777500141.16 total 494.04 last 30.56) - open/close 9256 (time 1777500151.79 total 504.67 last 24.56) - open/close 9556 (time 1777500163.89 total 516.77 last 24.79) - open/close 9817 (time 1777500174.57 total 527.45 last 24.44) - open/close 10000 (time 1777500180.15 total 533.02 last 32.81) - open/close 10332 (time 1777500190.15 total 543.03 last 33.19) - open/close 10633 (time 1777500201.14 total 554.02 last 27.39) - open/close 10925 (time 1777500211.15 total 564.03 last 29.17) - open/close 11318 (time 1777500221.15 total 574.03 last 39.29) - open/close 12314 (time 1777500231.16 total 584.03 last 99.55) - open/close 13388 (time 1777500241.18 total 594.05 last 107.22) - open/close 14506 (time 1777500251.18 total 604.05 last 111.76) - open/close 15653 (time 1777500261.19 total 614.06 last 114.62) - open/close 16861 (time 1777500271.19 total 624.07 last 120.74) - open/close 17899 (time 1777500281.20 total 634.08 last 103.71) - open/close 19060 (time 1777500291.20 total 644.08 last 116.06) - open/close 20000 (time 1777500299.65 total 652.53 last 111.24) - open/close 21093 (time 1777500309.66 total 662.53 last 109.26) - open/close 22246 (time 1777500319.66 total 672.54 last 115.23) - open/close 23490 (time 1777500329.67 total 682.54 last 124.37) - open/close 24695 (time 1777500339.67 total 692.54 last 120.47) - open/close 25857 (time 1777500349.68 total 702.55 last 116.11) - open/close 26997 (time 1777500359.68 total 712.56 last 113.91) - open/close 28080 (time 1777500369.69 total 722.56 last 108.25) - open/close 29367 (time 1777500379.70 total 732.57 last 128.61) - open/close 30000 (time 1777500384.24 total 737.12 last 139.18) - open/close 31414 (time 1777500394.25 total 747.12 last 141.39) - open/close 32736 (time 1777500404.25 total 757.12 last 132.17) - open/close 34086 (time 1777500414.25 total 767.12 last 134.98) - open/close 35281 (time 1777500424.25 total 777.13 last 119.44) - open/close 36939 (time 1777500434.26 total 787.13 last 165.76) - open/close 38769 (time 1777500444.26 total 797.13 last 182.99) - open/close 40000 (time 1777500451.45 total 804.33 last 171.07) - open/close 42128 (time 1777500461.46 total 814.33 last 212.71) - open/close 44026 (time 1777500471.46 total 824.33 last 189.78) - open/close 45953 (time 1777500481.46 total 834.34 last 192.58) - open/close 47681 (time 1777500491.47 total 844.34 last 172.77) - open/close 49604 (time 1777500501.47 total 854.35 last 192.20) total: 50000 open/close in 856.54 seconds: 58.37 ops/second - unlinked 0 (time 1777500505 ; total 0 ; last 0) - unlinked 10000 (time 1777500545 ; total 40 ; last 40) - unlinked 20000 (time 1777500583 ; total 78 ; last 38) - unlinked 30000 (time 1777500616 ; total 111 ; last 33) - unlinked 40000 (time 1777500648 ; total 143 ; last 32) total: 50000 unlinks in 173 seconds: 289.017334 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: modules unloaded. pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2