== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:02:15 (1787842935) start mds service on oleg453-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg453-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 313 (time 1787843023.21 total 10.01 last 31.27) - open/close 510 (time 1787843034.05 total 20.85 last 18.17) - open/close 734 (time 1787843045.50 total 32.30 last 19.56) - open/close 883 (time 1787843055.51 total 42.31 last 14.89) - open/close 1133 (time 1787843065.52 total 52.32 last 24.96) - open/close 1368 (time 1787843075.66 total 62.46 last 23.17) - open/close 1633 (time 1787843086.53 total 73.33 last 24.38) - open/close 1907 (time 1787843097.30 total 84.10 last 25.46) - open/close 2175 (time 1787843108.12 total 94.92 last 24.75) - open/close 2450 (time 1787843119.11 total 105.91 last 25.03) - open/close 2642 (time 1787843129.13 total 115.92 last 19.17) - open/close 2773 (time 1787843139.27 total 126.07 last 12.91) - open/close 2835 (time 1787843149.27 total 136.07 last 6.20) - open/close 2952 (time 1787843159.34 total 146.14 last 11.63) - open/close 3051 (time 1787843170.33 total 157.13 last 9.01) - open/close 3140 (time 1787843181.20 total 168.00 last 8.18) - open/close 3240 (time 1787843194.19 total 180.99 last 7.70) - open/close 3319 (time 1787843204.43 total 191.23 last 7.72) - open/close 3427 (time 1787843214.43 total 201.23 last 10.79) - open/close 3518 (time 1787843224.93 total 211.73 last 8.67) - open/close 3694 (time 1787843234.93 total 221.73 last 17.60) - open/close 3826 (time 1787843244.95 total 231.75 last 13.18) - open/close 4045 (time 1787843255.62 total 242.42 last 20.51) - open/close 4177 (time 1787843265.63 total 252.43 last 13.19) - open/close 4297 (time 1787843276.50 total 263.30 last 11.04) - open/close 4496 (time 1787843287.33 total 274.13 last 18.38) - open/close 4681 (time 1787843297.35 total 284.15 last 18.46) - open/close 4814 (time 1787843308.13 total 294.93 last 12.34) - open/close 5054 (time 1787843318.14 total 304.94 last 23.96) - open/close 5175 (time 1787843329.10 total 315.90 last 11.04) - open/close 5418 (time 1787843339.99 total 326.79 last 22.33) - open/close 5560 (time 1787843350.00 total 336.80 last 14.19) - open/close 5837 (time 1787843360.02 total 346.82 last 27.63) - open/close 6125 (time 1787843370.36 total 357.16 last 27.85) - open/close 6252 (time 1787843380.48 total 367.28 last 12.56) - open/close 6439 (time 1787843391.00 total 377.80 last 17.77) - open/close 6580 (time 1787843401.55 total 388.35 last 13.37) - open/close 6828 (time 1787843411.56 total 398.36 last 24.77) - open/close 6934 (time 1787843422.02 total 408.82 last 10.13) - open/close 7169 (time 1787843433.67 total 420.47 last 20.18) - open/close 7370 (time 1787843443.72 total 430.52 last 19.99) - open/close 7456 (time 1787843454.36 total 441.16 last 8.09) - open/close 7529 (time 1787843466.26 total 453.06 last 6.14) - open/close 7731 (time 1787843476.94 total 463.74 last 18.90) - open/close 8030 (time 1787843486.98 total 473.78 last 29.79) - open/close 8255 (time 1787843497.43 total 484.23 last 21.54) - open/close 8506 (time 1787843508.44 total 495.24 last 22.79) - open/close 8561 (time 1787843521.35 total 508.15 last 4.26) - open/close 8682 (time 1787843531.67 total 518.47 last 11.72) - open/close 8874 (time 1787843541.67 total 528.47 last 19.19) - open/close 8996 (time 1787843552.48 total 539.28 last 11.28) - open/close 9155 (time 1787843563.03 total 549.83 last 15.07) - open/close 9409 (time 1787843573.04 total 559.84 last 25.38) - open/close 9636 (time 1787843583.77 total 570.57 last 21.16) - open/close 9861 (time 1787843596.03 total 582.83 last 18.35) - open/close 10000 (time 1787843605.80 total 592.60 last 14.23) - open/close 10095 (time 1787843615.81 total 602.61 last 9.49) - open/close 10196 (time 1787843627.37 total 614.17 last 8.74) - open/close 10276 (time 1787843637.37 total 624.17 last 8.00) - open/close 10442 (time 1787843647.38 total 634.18 last 16.58) - open/close 10577 (time 1787843658.38 total 645.18 last 12.27) - open/close 10792 (time 1787843668.74 total 655.54 last 20.75) - open/close 10970 (time 1787843678.75 total 665.55 last 17.78) - open/close 11121 (time 1787843689.02 total 675.82 last 14.70) - open/close 11187 (time 1787843699.04 total 685.84 last 6.59) - open/close 11392 (time 1787843709.66 total 696.46 last 19.32) - open/close 11484 (time 1787843720.91 total 707.71 last 8.18) - open/close 11629 (time 1787843731.01 total 717.81 last 14.35) - open/close 11814 (time 1787843741.98 total 728.78 last 16.87) - open/close 12001 (time 1787843752.40 total 739.20 last 17.95) - open/close 12151 (time 1787843763.44 total 750.24 last 13.59) - open/close 12324 (time 1787843774.23 total 761.03 last 16.03) - open/close 12478 (time 1787843784.84 total 771.64 last 14.51) - open/close 12685 (time 1787843795.66 total 782.46 last 19.13) - open/close 12742 (time 1787843805.67 total 792.47 last 5.70) - open/close 12792 (time 1787843816.45 total 803.25 last 4.64) - open/close 12959 (time 1787843826.65 total 813.45 last 16.36) - open/close 13224 (time 1787843836.99 total 823.79 last 25.63) - open/close 13345 (time 1787843847.82 total 834.62 last 11.18) - open/close 13496 (time 1787843858.19 total 844.99 last 14.56) - open/close 13654 (time 1787843868.52 total 855.32 last 15.29) - open/close 13783 (time 1787843879.49 total 866.29 last 11.76) - open/close 13910 (time 1787843889.96 total 876.76 last 12.13) - open/close 14015 (time 1787843900.50 total 887.29 last 9.97) - open/close 14266 (time 1787843910.52 total 897.32 last 25.03) - open/close 14427 (time 1787843920.69 total 907.49 last 15.84) - open/close 14526 (time 1787843931.60 total 918.40 last 9.07) - open/close 14631 (time 1787843942.55 total 929.35 last 9.59) - open/close 14797 (time 1787843952.75 total 939.55 last 16.27) - open/close 14893 (time 1787843963.09 total 949.89 last 9.28) - open/close 15099 (time 1787843973.72 total 960.52 last 19.38) - open/close 15153 (time 1787843984.80 total 971.60 last 4.87) - open/close 15303 (time 1787843996.87 total 983.67 last 12.42) - open/close 15478 (time 1787844009.70 total 996.50 last 13.65) - open/close 15540 (time 1787844019.71 total 1006.51 last 6.19) - open/close 15700 (time 1787844029.71 total 1016.51 last 15.99) - open/close 15915 (time 1787844039.94 total 1026.74 last 21.04) - open/close 16097 (time 1787844049.94 total 1036.74 last 18.19) - open/close 16344 (time 1787844059.95 total 1046.75 last 24.69) - open/close 16538 (time 1787844070.03 total 1056.83 last 19.24) - open/close 16732 (time 1787844080.03 total 1066.83 last 19.39) - open/close 17444 (time 1787844090.04 total 1076.84 last 71.16) - open/close 18208 (time 1787844100.04 total 1086.84 last 76.39) - open/close 18902 (time 1787844110.05 total 1096.85 last 69.35) - open/close 19594 (time 1787844120.06 total 1106.86 last 69.14) - open/close 20000 (time 1787844125.49 total 1112.29 last 74.76) - open/close 20789 (time 1787844135.50 total 1122.30 last 78.82) - open/close 21575 (time 1787844145.50 total 1132.30 last 78.58) - open/close 22341 (time 1787844155.51 total 1142.31 last 76.52) - open/close 23050 (time 1787844165.51 total 1152.31 last 70.88) - open/close 23809 (time 1787844175.52 total 1162.32 last 75.88) - open/close 24567 (time 1787844185.52 total 1172.32 last 75.77) - open/close 25331 (time 1787844195.52 total 1182.32 last 76.38) - open/close 26135 (time 1787844205.53 total 1192.33 last 80.34) - open/close 26856 (time 1787844215.54 total 1202.34 last 72.01) - open/close 27624 (time 1787844225.54 total 1212.34 last 76.79) - open/close 28380 (time 1787844235.55 total 1222.35 last 75.57) - open/close 29162 (time 1787844245.55 total 1232.35 last 78.15) - open/close 29949 (time 1787844255.58 total 1242.38 last 78.48) - open/close 30701 (time 1787844265.59 total 1252.39 last 75.13) - open/close 31475 (time 1787844275.60 total 1262.40 last 77.33) - open/close 32210 (time 1787844285.61 total 1272.41 last 73.41) - open/close 32942 (time 1787844295.62 total 1282.42 last 73.16) - open/close 33689 (time 1787844305.63 total 1292.43 last 74.61) - open/close 34455 (time 1787844315.63 total 1302.43 last 76.59) - open/close 35162 (time 1787844325.65 total 1312.45 last 70.59) - open/close 35936 (time 1787844335.67 total 1322.47 last 77.19) - open/close 36675 (time 1787844345.67 total 1332.47 last 73.89) - open/close 37452 (time 1787844355.68 total 1342.48 last 77.69) - open/close 38162 (time 1787844365.68 total 1352.48 last 70.99) - open/close 38890 (time 1787844375.68 total 1362.48 last 72.77) - open/close 39644 (time 1787844385.68 total 1372.48 last 75.40) - open/close 40000 (time 1787844390.19 total 1376.99 last 78.94) - open/close 40750 (time 1787844400.19 total 1386.99 last 74.98) - open/close 41408 (time 1787844410.20 total 1397.00 last 65.77) - open/close 42003 (time 1787844420.21 total 1407.01 last 59.45) - open/close 42662 (time 1787844430.22 total 1417.02 last 65.84) - open/close 43424 (time 1787844440.22 total 1427.02 last 76.15) - open/close 44180 (time 1787844450.22 total 1437.02 last 75.60) - open/close 44923 (time 1787844460.23 total 1447.03 last 74.23) - open/close 45626 (time 1787844470.25 total 1457.05 last 70.21) - open/close 46338 (time 1787844480.25 total 1467.05 last 71.14) - open/close 47057 (time 1787844490.26 total 1477.06 last 71.84) - open/close 47778 (time 1787844500.27 total 1487.07 last 72.08) - open/close 48544 (time 1787844510.27 total 1497.07 last 76.54) - open/close 49316 (time 1787844520.28 total 1507.08 last 77.14) total: 50000 open/close in 1516.13 seconds: 32.98 ops/second - unlinked 0 (time 1787844533 ; total 0 ; last 0) - unlinked 10000 (time 1787844681 ; total 148 ; last 148) - unlinked 20000 (time 1787844829 ; total 296 ; last 148) - unlinked 30000 (time 1787844976 ; total 443 ; last 147) - unlinked 40000 (time 1787845119 ; total 586 ; last 143) total: 50000 unlinks in 729 seconds: 68.587105 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2