== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:21:47 (1786749707) start mds service on oleg643-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing load_modules_local oleg643-server: Loading modules from /home/green/git/lustre-release/lustre oleg643-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg643-server: detected 4 online CPUs by sysfs oleg643-server: MODOPTS_LIBCFS= oleg643-server: Force libcfs to create 2 CPU partitions oleg643-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg643-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg643-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1786749789.24 total 10.92 last 23.53) - open/close 501 (time 1786749799.25 total 20.92 last 24.39) - open/close 669 (time 1786749809.25 total 30.93 last 16.80) - open/close 834 (time 1786749819.83 total 41.51 last 15.59) - open/close 1098 (time 1786749830.78 total 52.46 last 24.11) - open/close 1289 (time 1786749840.80 total 62.48 last 19.07) - open/close 1524 (time 1786749850.80 total 72.48 last 23.48) - open/close 1736 (time 1786749861.44 total 83.11 last 19.94) - open/close 1941 (time 1786749872.31 total 93.98 last 18.86) - open/close 2189 (time 1786749882.32 total 104.00 last 24.77) - open/close 2441 (time 1786749892.33 total 114.01 last 25.17) - open/close 2639 (time 1786749902.69 total 124.37 last 19.11) - open/close 2940 (time 1786749912.70 total 134.37 last 30.09) - open/close 3186 (time 1786749922.88 total 144.56 last 24.16) - open/close 3481 (time 1786749932.89 total 154.57 last 29.45) - open/close 3726 (time 1786749942.90 total 164.58 last 24.47) - open/close 3948 (time 1786749953.88 total 175.55 last 20.23) - open/close 4198 (time 1786749964.14 total 185.82 last 24.35) - open/close 4485 (time 1786749974.21 total 195.89 last 28.51) - open/close 4680 (time 1786749984.21 total 205.89 last 19.49) - open/close 4993 (time 1786749994.23 total 215.90 last 31.26) - open/close 5247 (time 1786750004.23 total 225.91 last 25.38) - open/close 5455 (time 1786750015.28 total 236.96 last 18.83) - open/close 5608 (time 1786750026.18 total 247.86 last 14.04) - open/close 5863 (time 1786750036.48 total 258.15 last 24.77) - open/close 6105 (time 1786750046.54 total 268.22 last 24.05) - open/close 6364 (time 1786750056.95 total 278.63 last 24.87) - open/close 6625 (time 1786750067.59 total 289.27 last 24.53) - open/close 6899 (time 1786750077.60 total 299.27 last 27.39) - open/close 7124 (time 1786750088.58 total 310.25 last 20.49) - open/close 7356 (time 1786750098.62 total 320.30 last 23.10) - open/close 7552 (time 1786750109.59 total 331.27 last 17.86) - open/close 7771 (time 1786750120.13 total 341.81 last 20.79) - open/close 7967 (time 1786750130.49 total 352.17 last 18.91) - open/close 8021 (time 1786750140.86 total 362.54 last 5.21) - open/close 8215 (time 1786750151.90 total 373.58 last 17.56) - open/close 8419 (time 1786750161.91 total 383.59 last 20.39) - open/close 8628 (time 1786750171.92 total 393.60 last 20.88) - open/close 8846 (time 1786750183.29 total 404.97 last 19.16) - open/close 9075 (time 1786750194.24 total 415.92 last 20.92) - open/close 9242 (time 1786750204.94 total 426.62 last 15.60) - open/close 9487 (time 1786750214.95 total 436.63 last 24.49) - open/close 9581 (time 1786750225.28 total 446.96 last 9.09) - open/close 9715 (time 1786750235.84 total 457.52 last 12.69) - open/close 9826 (time 1786750245.86 total 467.54 last 11.08) - open/close 10000 (time 1786750253.62 total 475.30 last 22.42) - open/close 10252 (time 1786750264.72 total 486.39 last 22.71) - open/close 10508 (time 1786750274.72 total 496.40 last 25.60) - open/close 10763 (time 1786750285.07 total 506.75 last 24.63) - open/close 11007 (time 1786750295.81 total 517.49 last 22.72) - open/close 11270 (time 1786750306.10 total 527.77 last 25.58) - open/close 11506 (time 1786750318.66 total 540.34 last 18.78) - open/close 11781 (time 1786750329.22 total 550.89 last 26.05) - open/close 12037 (time 1786750340.87 total 562.55 last 21.96) - open/close 12302 (time 1786750351.94 total 573.62 last 23.95) - open/close 12557 (time 1786750361.95 total 583.63 last 25.47) - open/close 12816 (time 1786750372.60 total 594.28 last 24.31) - open/close 13053 (time 1786750383.16 total 604.84 last 22.45) - open/close 13300 (time 1786750393.99 total 615.67 last 22.81) - open/close 13580 (time 1786750404.01 total 625.69 last 27.94) - open/close 13786 (time 1786750414.01 total 635.69 last 20.59) - open/close 14110 (time 1786750424.89 total 646.57 last 29.78) - open/close 14360 (time 1786750435.07 total 656.74 last 24.57) - open/close 14602 (time 1786750445.46 total 667.13 last 23.29) - open/close 14810 (time 1786750455.46 total 677.14 last 20.79) - open/close 15061 (time 1786750465.85 total 687.53 last 24.16) - open/close 15320 (time 1786750476.75 total 698.42 last 23.78) - open/close 15569 (time 1786750486.75 total 708.43 last 24.88) - open/close 15788 (time 1786750497.08 total 718.76 last 21.20) - open/close 16068 (time 1786750507.29 total 728.96 last 27.44) - open/close 16192 (time 1786750519.81 total 741.49 last 9.90) - open/close 16425 (time 1786750529.82 total 751.50 last 23.29) - open/close 16566 (time 1786750540.67 total 762.35 last 12.99) - open/close 16799 (time 1786750551.40 total 773.08 last 21.72) - open/close 17013 (time 1786750562.36 total 784.03 last 19.53) - open/close 17233 (time 1786750572.36 total 794.04 last 21.99) - open/close 17484 (time 1786750583.37 total 805.04 last 22.81) - open/close 17684 (time 1786750593.38 total 815.06 last 19.97) - open/close 17918 (time 1786750603.98 total 825.65 last 22.08) - open/close 18180 (time 1786750614.78 total 836.45 last 24.26) - open/close 18451 (time 1786750624.79 total 846.47 last 27.06) - open/close 18693 (time 1786750635.15 total 856.83 last 23.36) - open/close 18968 (time 1786750645.15 total 866.83 last 27.49) - open/close 19169 (time 1786750655.16 total 876.84 last 20.09) - open/close 19407 (time 1786750666.13 total 887.81 last 21.69) - open/close 19686 (time 1786750677.17 total 898.85 last 25.28) - open/close 19970 (time 1786750687.17 total 908.85 last 28.40) - open/close 20234 (time 1786750697.18 total 918.86 last 26.39) - open/close 20480 (time 1786750707.18 total 928.86 last 24.60) - open/close 21179 (time 1786750717.18 total 938.86 last 69.89) - open/close 22010 (time 1786750727.18 total 948.86 last 83.10) - open/close 22899 (time 1786750737.20 total 958.88 last 88.73) - open/close 23594 (time 1786750747.20 total 968.88 last 69.50) - open/close 24367 (time 1786750757.20 total 978.88 last 77.27) - open/close 25171 (time 1786750767.21 total 988.89 last 80.33) - open/close 25907 (time 1786750777.22 total 998.90 last 73.54) - open/close 26765 (time 1786750787.22 total 1008.90 last 85.79) - open/close 27571 (time 1786750797.23 total 1018.90 last 80.54) - open/close 28344 (time 1786750807.23 total 1028.91 last 77.26) - open/close 29249 (time 1786750817.24 total 1038.91 last 90.47) - open/close 30000 (time 1786750825.52 total 1047.20 last 90.65) - open/close 30811 (time 1786750835.52 total 1057.20 last 81.09) - open/close 31673 (time 1786750845.52 total 1067.20 last 86.19) - open/close 32509 (time 1786750855.52 total 1077.20 last 83.60) - open/close 33393 (time 1786750865.53 total 1087.21 last 88.34) - open/close 34192 (time 1786750875.54 total 1097.21 last 79.85) - open/close 35003 (time 1786750885.55 total 1107.22 last 81.02) - open/close 35865 (time 1786750895.56 total 1117.24 last 86.10) - open/close 36720 (time 1786750905.56 total 1127.24 last 85.47) - open/close 37493 (time 1786750915.57 total 1137.24 last 77.26) - open/close 38241 (time 1786750925.57 total 1147.25 last 74.79) - open/close 39096 (time 1786750935.58 total 1157.25 last 85.44) - open/close 39931 (time 1786750945.58 total 1167.26 last 83.46) - open/close 40759 (time 1786750955.59 total 1177.26 last 82.75) - open/close 41604 (time 1786750965.59 total 1187.27 last 84.47) - open/close 42449 (time 1786750975.60 total 1197.28 last 84.42) - open/close 43320 (time 1786750985.68 total 1207.36 last 86.40) - open/close 44149 (time 1786750995.70 total 1217.38 last 82.73) - open/close 44955 (time 1786751005.71 total 1227.39 last 80.52) - open/close 45629 (time 1786751015.71 total 1237.39 last 67.38) - open/close 46457 (time 1786751025.72 total 1247.40 last 82.74) - open/close 47247 (time 1786751035.72 total 1257.40 last 78.97) - open/close 48074 (time 1786751045.73 total 1267.41 last 82.64) - open/close 48847 (time 1786751055.74 total 1277.41 last 77.26) - open/close 49543 (time 1786751065.74 total 1287.42 last 69.58) total: 50000 open/close in 1292.94 seconds: 38.67 ops/second - unlinked 0 (time 1786751075 ; total 0 ; last 0) - unlinked 10000 (time 1786751199 ; total 124 ; last 124) - unlinked 20000 (time 1786751323 ; total 248 ; last 124) - unlinked 30000 (time 1786751457 ; total 382 ; last 134) - unlinked 40000 (time 1786751593 ; total 518 ; last 136) total: 50000 unlinks in 647 seconds: 77.279755 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg643-server unloading modules via unload_modules_local on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing unload_modules_local oleg643-server: modules unloaded. pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2