== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:10:00 (1786482600) start mds service on oleg448-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg448-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1786482665.62 total 10.22 last 19.86) - open/close 399 (time 1786482675.67 total 20.27 last 19.50) - open/close 587 (time 1786482685.69 total 30.29 last 18.76) - open/close 760 (time 1786482696.35 total 40.95 last 16.23) - open/close 942 (time 1786482707.34 total 51.94 last 16.55) - open/close 1078 (time 1786482717.53 total 62.13 last 13.35) - open/close 1209 (time 1786482728.54 total 73.15 last 11.89) - open/close 1319 (time 1786482742.16 total 86.76 last 8.08) - open/close 1457 (time 1786482752.17 total 96.77 last 13.78) - open/close 1561 (time 1786482762.84 total 107.44 last 9.75) - open/close 1694 (time 1786482772.85 total 117.45 last 13.29) - open/close 1759 (time 1786482782.93 total 127.53 last 6.45) - open/close 1839 (time 1786482793.23 total 137.83 last 7.76) - open/close 2001 (time 1786482804.06 total 148.66 last 14.96) - open/close 2133 (time 1786482814.06 total 158.66 last 13.20) - open/close 2185 (time 1786482824.08 total 168.68 last 5.19) - open/close 2327 (time 1786482834.27 total 178.87 last 13.94) - open/close 2462 (time 1786482844.30 total 188.90 last 13.45) - open/close 2650 (time 1786482854.56 total 199.16 last 18.33) - open/close 2792 (time 1786482867.69 total 212.29 last 10.82) - open/close 3039 (time 1786482877.78 total 222.38 last 24.46) - open/close 3197 (time 1786482887.79 total 232.39 last 15.79) - open/close 3319 (time 1786482898.25 total 242.85 last 11.66) - open/close 3564 (time 1786482910.10 total 254.70 last 20.68) - open/close 3824 (time 1786482921.05 total 265.65 last 23.75) - open/close 3903 (time 1786482931.41 total 276.01 last 7.62) - open/close 4000 (time 1786482941.97 total 286.57 last 9.19) - open/close 4086 (time 1786482954.34 total 298.94 last 6.95) - open/close 4323 (time 1786482965.34 total 309.94 last 21.53) - open/close 4495 (time 1786482975.46 total 320.06 last 16.99) - open/close 4640 (time 1786482985.47 total 330.07 last 14.49) - open/close 4942 (time 1786482995.81 total 340.41 last 29.20) - open/close 5216 (time 1786483006.68 total 351.28 last 25.22) - open/close 5555 (time 1786483016.68 total 361.28 last 33.90) - open/close 5838 (time 1786483026.70 total 371.30 last 28.23) - open/close 6164 (time 1786483036.71 total 381.32 last 32.57) - open/close 6460 (time 1786483047.37 total 391.98 last 27.77) - open/close 6759 (time 1786483057.38 total 401.98 last 29.90) - open/close 7054 (time 1786483067.61 total 412.21 last 28.83) - open/close 7387 (time 1786483077.84 total 422.44 last 32.54) - open/close 7577 (time 1786483089.05 total 433.65 last 16.96) - open/close 7934 (time 1786483099.54 total 444.14 last 34.01) - open/close 8085 (time 1786483109.59 total 454.19 last 15.03) - open/close 8352 (time 1786483119.70 total 464.30 last 26.40) - open/close 8640 (time 1786483129.71 total 474.31 last 28.78) - open/close 8940 (time 1786483139.80 total 484.40 last 29.74) - open/close 9265 (time 1786483149.97 total 494.57 last 31.94) - open/close 9604 (time 1786483159.98 total 504.58 last 33.88) - open/close 9914 (time 1786483172.81 total 517.41 last 24.15) - open/close 10000 (time 1786483175.63 total 520.24 last 30.48) - open/close 10332 (time 1786483185.64 total 530.24 last 33.20) - open/close 10623 (time 1786483195.64 total 540.24 last 29.08) - open/close 11641 (time 1786483205.64 total 550.24 last 101.78) - open/close 12824 (time 1786483215.65 total 560.25 last 118.25) - open/close 14034 (time 1786483225.66 total 570.26 last 120.85) - open/close 15234 (time 1786483235.66 total 580.26 last 120.00) - open/close 16442 (time 1786483245.67 total 590.27 last 120.73) - open/close 17658 (time 1786483255.67 total 600.27 last 121.57) - open/close 18923 (time 1786483265.68 total 610.28 last 126.41) - open/close 20000 (time 1786483273.89 total 618.49 last 131.03) - open/close 21262 (time 1786483283.90 total 628.50 last 126.09) - open/close 22477 (time 1786483293.91 total 638.51 last 121.44) - open/close 23871 (time 1786483303.91 total 648.51 last 139.37) - open/close 25274 (time 1786483313.91 total 658.51 last 140.28) - open/close 26628 (time 1786483323.91 total 668.52 last 135.36) - open/close 27860 (time 1786483333.92 total 678.52 last 123.19) - open/close 29082 (time 1786483343.92 total 688.52 last 122.12) - open/close 30000 (time 1786483350.90 total 695.50 last 131.51) - open/close 31378 (time 1786483360.91 total 705.51 last 137.72) - open/close 32968 (time 1786483370.91 total 715.51 last 158.94) - open/close 34679 (time 1786483380.91 total 725.51 last 171.10) - open/close 36356 (time 1786483390.91 total 735.52 last 167.65) - open/close 37862 (time 1786483400.92 total 745.52 last 150.57) - open/close 39333 (time 1786483410.92 total 755.52 last 147.02) - open/close 40000 (time 1786483415.83 total 760.43 last 136.03) - open/close 41474 (time 1786483425.83 total 770.43 last 147.34) - open/close 42838 (time 1786483435.84 total 780.44 last 136.26) - open/close 44209 (time 1786483445.84 total 790.44 last 137.06) - open/close 45795 (time 1786483455.84 total 800.44 last 158.58) - open/close 47288 (time 1786483465.84 total 810.44 last 149.29) - open/close 48568 (time 1786483475.85 total 820.45 last 127.98) - open/close 49846 (time 1786483485.85 total 830.45 last 127.79) total: 50000 open/close in 831.59 seconds: 60.13 ops/second - unlinked 0 (time 1786483489 ; total 0 ; last 0) - unlinked 10000 (time 1786483542 ; total 53 ; last 53) - unlinked 20000 (time 1786483576 ; total 87 ; last 34) - unlinked 30000 (time 1786483610 ; total 121 ; last 34) - unlinked 40000 (time 1786483640 ; total 151 ; last 30) total: 50000 unlinks in 186 seconds: 268.817200 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2