== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:14:08 (1786965248) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 174 (time 1786965343.67 total 10.01 last 17.38) - open/close 390 (time 1786965353.68 total 20.02 last 21.59) - open/close 585 (time 1786965364.16 total 30.50 last 18.60) - open/close 799 (time 1786965374.90 total 41.24 last 19.93) - open/close 973 (time 1786965394.29 total 60.63 last 8.97) - open/close 1125 (time 1786965404.31 total 70.65 last 15.16) - open/close 1272 (time 1786965414.31 total 80.65 last 14.70) - open/close 1381 (time 1786965426.11 total 92.45 last 9.24) - open/close 1599 (time 1786965436.74 total 103.08 last 20.50) - open/close 1704 (time 1786965448.60 total 114.95 last 8.85) - open/close 1903 (time 1786965459.51 total 125.85 last 18.25) - open/close 2004 (time 1786965469.54 total 135.89 last 10.07) - open/close 2128 (time 1786965479.56 total 145.90 last 12.38) - open/close 2332 (time 1786965489.56 total 155.90 last 20.40) - open/close 2518 (time 1786965499.56 total 165.91 last 18.59) - open/close 2728 (time 1786965510.06 total 176.40 last 20.01) - open/close 2986 (time 1786965520.25 total 186.59 last 25.31) - open/close 3216 (time 1786965530.44 total 196.78 last 22.58) - open/close 3421 (time 1786965540.74 total 207.08 last 19.90) - open/close 3568 (time 1786965551.36 total 217.71 last 13.84) - open/close 3774 (time 1786965561.38 total 227.72 last 20.57) - open/close 3962 (time 1786965571.52 total 237.86 last 18.55) - open/close 4192 (time 1786965581.52 total 247.86 last 23.00) - open/close 4368 (time 1786965591.64 total 257.98 last 17.39) - open/close 4528 (time 1786965602.05 total 268.39 last 15.37) - open/close 4753 (time 1786965612.65 total 278.99 last 21.22) - open/close 4957 (time 1786965622.67 total 289.01 last 20.37) - open/close 5153 (time 1786965632.70 total 299.04 last 19.54) - open/close 5323 (time 1786965642.72 total 309.07 last 16.95) - open/close 5436 (time 1786965653.54 total 319.88 last 10.45) - open/close 5716 (time 1786965663.54 total 329.88 last 28.00) - open/close 5964 (time 1786965674.48 total 340.82 last 22.67) - open/close 6129 (time 1786965684.90 total 351.24 last 15.83) - open/close 6342 (time 1786965695.31 total 361.65 last 20.47) - open/close 6604 (time 1786965706.28 total 372.62 last 23.88) - open/close 6793 (time 1786965716.61 total 382.96 last 18.28) - open/close 6875 (time 1786965727.28 total 393.62 last 7.69) - open/close 7013 (time 1786965737.95 total 404.29 last 12.94) - open/close 7218 (time 1786965748.83 total 415.17 last 18.83) - open/close 7378 (time 1786965760.31 total 426.65 last 13.94) - open/close 7597 (time 1786965770.31 total 436.66 last 21.89) - open/close 7789 (time 1786965780.61 total 446.95 last 18.65) - open/close 8006 (time 1786965791.08 total 457.42 last 20.72) - open/close 8192 (time 1786965801.64 total 467.99 last 17.61) - open/close 8369 (time 1786965812.57 total 478.91 last 16.20) - open/close 8584 (time 1786965822.57 total 488.92 last 21.49) - open/close 8809 (time 1786965833.51 total 499.85 last 20.58) - open/close 9052 (time 1786965843.51 total 509.86 last 24.28) - open/close 9227 (time 1786965854.22 total 520.57 last 16.34) - open/close 9513 (time 1786965866.85 total 533.19 last 22.65) - open/close 9668 (time 1786965877.30 total 543.64 last 14.84) - open/close 9911 (time 1786965887.31 total 553.65 last 24.26) - open/close 10000 (time 1786965891.18 total 557.52 last 23.02) - open/close 10185 (time 1786965901.20 total 567.55 last 18.46) - open/close 10405 (time 1786965911.96 total 578.30 last 20.46) - open/close 10650 (time 1786965921.96 total 588.30 last 24.49) - open/close 10887 (time 1786965932.97 total 599.31 last 21.53) - open/close 10921 (time 1786965944.25 total 610.59 last 3.02) - open/close 11001 (time 1786965954.76 total 621.10 last 7.61) - open/close 11220 (time 1786965965.61 total 631.95 last 20.18) - open/close 11442 (time 1786965975.84 total 642.18 last 21.71) - open/close 11624 (time 1786965985.84 total 652.19 last 18.19) - open/close 11898 (time 1786965995.85 total 662.19 last 27.39) - open/close 12127 (time 1786966005.85 total 672.19 last 22.89) - open/close 12289 (time 1786966017.85 total 684.19 last 13.51) - open/close 12503 (time 1786966033.52 total 699.86 last 13.65) - open/close 12664 (time 1786966046.33 total 712.68 last 12.56) - open/close 12780 (time 1786966056.62 total 722.96 last 11.28) - open/close 12934 (time 1786966066.91 total 733.26 last 14.96) - open/close 13151 (time 1786966077.30 total 743.64 last 20.89) - open/close 13426 (time 1786966088.17 total 754.51 last 25.30) - open/close 13659 (time 1786966098.28 total 764.62 last 23.05) - open/close 13833 (time 1786966108.30 total 774.64 last 17.37) - open/close 14017 (time 1786966118.31 total 784.65 last 18.39) - open/close 14139 (time 1786966128.32 total 794.67 last 12.18) - open/close 14363 (time 1786966138.53 total 804.87 last 21.95) - open/close 14610 (time 1786966148.53 total 814.88 last 24.69) - open/close 14836 (time 1786966158.54 total 824.89 last 22.58) - open/close 15070 (time 1786966168.55 total 834.89 last 23.40) - open/close 15306 (time 1786966178.55 total 844.89 last 23.59) - open/close 15537 (time 1786966188.79 total 855.13 last 22.56) - open/close 15758 (time 1786966198.81 total 865.15 last 22.05) - open/close 15921 (time 1786966209.12 total 875.46 last 15.82) - open/close 16131 (time 1786966219.18 total 885.52 last 20.88) - open/close 16386 (time 1786966229.19 total 895.53 last 25.47) - open/close 16632 (time 1786966239.20 total 905.54 last 24.56) - open/close 16841 (time 1786966249.32 total 915.67 last 20.65) - open/close 17048 (time 1786966259.34 total 925.68 last 20.66) - open/close 17290 (time 1786966270.12 total 936.46 last 22.46) - open/close 17474 (time 1786966280.24 total 946.58 last 18.18) - open/close 17631 (time 1786966290.40 total 956.74 last 15.45) - open/close 17782 (time 1786966301.22 total 967.57 last 13.95) - open/close 18149 (time 1786966311.23 total 977.57 last 36.69) - open/close 18910 (time 1786966321.23 total 987.57 last 76.05) - open/close 19683 (time 1786966331.25 total 997.59 last 77.19) - open/close 20000 (time 1786966335.29 total 1001.63 last 78.39) - open/close 20744 (time 1786966345.29 total 1011.63 last 74.40) - open/close 21503 (time 1786966355.31 total 1021.65 last 75.76) - open/close 22268 (time 1786966365.32 total 1031.66 last 76.44) - open/close 23015 (time 1786966375.33 total 1041.67 last 74.60) - open/close 23789 (time 1786966385.34 total 1051.68 last 77.32) - open/close 24495 (time 1786966395.35 total 1061.69 last 70.55) - open/close 25241 (time 1786966405.35 total 1071.69 last 74.58) - open/close 25940 (time 1786966415.36 total 1081.70 last 69.86) - open/close 26704 (time 1786966425.37 total 1091.71 last 76.32) - open/close 27400 (time 1786966435.37 total 1101.71 last 69.58) - open/close 28128 (time 1786966445.38 total 1111.72 last 72.77) - open/close 28977 (time 1786966455.39 total 1121.73 last 84.79) - open/close 29686 (time 1786966465.39 total 1131.74 last 70.86) - open/close 30000 (time 1786966470.41 total 1136.75 last 62.61) - open/close 30583 (time 1786966480.42 total 1146.76 last 58.26) - open/close 31215 (time 1786966490.46 total 1156.80 last 62.93) - open/close 31912 (time 1786966500.46 total 1166.80 last 69.70) - open/close 32611 (time 1786966510.46 total 1176.81 last 69.87) - open/close 33351 (time 1786966520.47 total 1186.81 last 73.98) - open/close 34047 (time 1786966530.48 total 1196.82 last 69.51) - open/close 34769 (time 1786966540.49 total 1206.83 last 72.15) - open/close 35369 (time 1786966550.49 total 1216.84 last 59.96) - open/close 36042 (time 1786966560.52 total 1226.86 last 67.15) - open/close 36844 (time 1786966570.52 total 1236.87 last 80.14) - open/close 37557 (time 1786966580.53 total 1246.87 last 71.24) - open/close 38285 (time 1786966590.54 total 1256.88 last 72.75) - open/close 39059 (time 1786966600.55 total 1266.89 last 77.32) - open/close 39775 (time 1786966610.57 total 1276.92 last 71.43) - open/close 40000 (time 1786966614.39 total 1280.73 last 58.96) - open/close 40692 (time 1786966624.40 total 1290.74 last 69.16) - open/close 41324 (time 1786966634.41 total 1300.75 last 63.11) - open/close 41920 (time 1786966644.41 total 1310.76 last 59.57) - open/close 42560 (time 1786966654.44 total 1320.78 last 63.84) - open/close 43099 (time 1786966664.46 total 1330.80 last 53.78) - open/close 43684 (time 1786966674.47 total 1340.81 last 58.48) - open/close 44256 (time 1786966684.48 total 1350.82 last 57.13) - open/close 44791 (time 1786966694.48 total 1360.82 last 53.49) - open/close 45458 (time 1786966704.49 total 1370.83 last 66.63) - open/close 46127 (time 1786966714.50 total 1380.84 last 66.82) - open/close 46811 (time 1786966724.52 total 1390.86 last 68.29) - open/close 47488 (time 1786966734.53 total 1400.87 last 67.64) - open/close 48206 (time 1786966744.54 total 1410.88 last 71.71) - open/close 48798 (time 1786966754.57 total 1420.92 last 59.00) - open/close 49256 (time 1786966764.58 total 1430.92 last 45.76) - open/close 49918 (time 1786966774.59 total 1440.94 last 66.12) total: 50000 open/close in 1442.22 seconds: 34.67 ops/second - unlinked 0 (time 1786966780 ; total 0 ; last 0) - unlinked 10000 (time 1786966942 ; total 162 ; last 162) - unlinked 20000 (time 1786967102 ; total 322 ; last 160) - unlinked 30000 (time 1786967269 ; total 489 ; last 167) - unlinked 40000 (time 1786967421 ; total 641 ; last 152) total: 50000 unlinks in 797 seconds: 62.735256 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2