== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:00:12 (1787122812) start mds service on oleg459-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg459-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 146 (time 1787122912.60 total 10.03 last 14.56) - open/close 312 (time 1787122922.77 total 20.20 last 16.33) - open/close 501 (time 1787122932.80 total 30.23 last 18.83) - open/close 734 (time 1787122943.15 total 40.58 last 22.52) - open/close 976 (time 1787122953.15 total 50.58 last 24.19) - open/close 1221 (time 1787122963.26 total 60.69 last 24.24) - open/close 1295 (time 1787122973.26 total 70.69 last 7.40) - open/close 1472 (time 1787122983.27 total 80.70 last 17.69) - open/close 1650 (time 1787122993.86 total 91.29 last 16.81) - open/close 1846 (time 1787123003.86 total 101.29 last 19.59) - open/close 2119 (time 1787123014.80 total 112.23 last 24.95) - open/close 2230 (time 1787123024.81 total 122.24 last 11.10) - open/close 2324 (time 1787123034.82 total 132.25 last 9.39) - open/close 2421 (time 1787123045.76 total 143.19 last 8.87) - open/close 2507 (time 1787123056.17 total 153.60 last 8.26) - open/close 2637 (time 1787123066.96 total 164.39 last 12.05) - open/close 2787 (time 1787123077.05 total 174.48 last 14.86) - open/close 2872 (time 1787123087.19 total 184.62 last 8.39) - open/close 3013 (time 1787123097.67 total 195.10 last 13.45) - open/close 3122 (time 1787123107.67 total 205.10 last 10.90) - open/close 3381 (time 1787123117.68 total 215.11 last 25.88) - open/close 3579 (time 1787123128.23 total 225.66 last 18.76) - open/close 3801 (time 1787123138.50 total 235.93 last 21.62) - open/close 4056 (time 1787123148.51 total 245.94 last 25.48) - open/close 4245 (time 1787123158.58 total 256.01 last 18.77) - open/close 4426 (time 1787123169.85 total 267.28 last 16.05) - open/close 4667 (time 1787123180.72 total 278.15 last 22.17) - open/close 4879 (time 1787123191.04 total 288.47 last 20.55) - open/close 5105 (time 1787123201.78 total 299.21 last 21.04) - open/close 5229 (time 1787123211.80 total 309.23 last 12.38) - open/close 5387 (time 1787123222.35 total 319.78 last 14.97) - open/close 5614 (time 1787123232.36 total 329.79 last 22.67) - open/close 5826 (time 1787123242.69 total 340.12 last 20.52) - open/close 6056 (time 1787123252.73 total 350.16 last 22.91) - open/close 6217 (time 1787123262.75 total 360.18 last 16.07) - open/close 6395 (time 1787123272.95 total 370.38 last 17.45) - open/close 6523 (time 1787123282.96 total 380.39 last 12.79) - open/close 6744 (time 1787123293.83 total 391.26 last 20.33) - open/close 6969 (time 1787123304.18 total 401.61 last 21.73) - open/close 7162 (time 1787123315.20 total 412.63 last 17.52) - open/close 7339 (time 1787123326.12 total 423.55 last 16.20) - open/close 7500 (time 1787123336.14 total 433.57 last 16.07) - open/close 7619 (time 1787123346.32 total 443.75 last 11.69) - open/close 7802 (time 1787123356.60 total 454.03 last 17.81) - open/close 7950 (time 1787123369.36 total 466.79 last 11.60) - open/close 8121 (time 1787123379.38 total 476.81 last 17.06) - open/close 8301 (time 1787123389.41 total 486.84 last 17.94) - open/close 8521 (time 1787123400.37 total 497.80 last 20.08) - open/close 8658 (time 1787123410.68 total 508.11 last 13.29) - open/close 8816 (time 1787123420.70 total 518.13 last 15.77) - open/close 8876 (time 1787123430.71 total 528.14 last 5.99) - open/close 8970 (time 1787123441.21 total 538.64 last 8.95) - open/close 9112 (time 1787123451.23 total 548.66 last 14.18) - open/close 9166 (time 1787123462.08 total 559.51 last 4.97) - open/close 9401 (time 1787123472.10 total 569.53 last 23.47) - open/close 9613 (time 1787123482.84 total 580.27 last 19.73) - open/close 9814 (time 1787123493.03 total 590.46 last 19.73) - open/close 9993 (time 1787123504.01 total 601.44 last 16.30) - open/close 10229 (time 1787123514.02 total 611.45 last 23.58) - open/close 10438 (time 1787123524.04 total 621.47 last 20.86) - open/close 10667 (time 1787123534.04 total 631.47 last 22.89) - open/close 10878 (time 1787123544.12 total 641.55 last 20.94) - open/close 11047 (time 1787123557.72 total 655.15 last 12.43) - open/close 11149 (time 1787123567.72 total 665.15 last 10.19) - open/close 11401 (time 1787123578.42 total 675.85 last 23.56) - open/close 11608 (time 1787123588.98 total 686.41 last 19.61) - open/close 11826 (time 1787123599.89 total 697.32 last 19.98) - open/close 12055 (time 1787123610.69 total 708.12 last 21.20) - open/close 12317 (time 1787123623.61 total 721.04 last 20.29) - open/close 12384 (time 1787123634.29 total 731.72 last 6.27) - open/close 12470 (time 1787123644.81 total 742.24 last 8.18) - open/close 12665 (time 1787123655.67 total 753.10 last 17.96) - open/close 12802 (time 1787123665.69 total 763.12 last 13.67) - open/close 13001 (time 1787123676.10 total 773.53 last 19.11) - open/close 13226 (time 1787123686.67 total 784.10 last 21.29) - open/close 13484 (time 1787123697.67 total 795.10 last 23.46) - open/close 13603 (time 1787123707.68 total 805.11 last 11.88) - open/close 13728 (time 1787123717.77 total 815.20 last 12.40) - open/close 13884 (time 1787123727.78 total 825.21 last 15.58) - open/close 14047 (time 1787123738.48 total 835.91 last 15.23) - open/close 14277 (time 1787123748.49 total 845.92 last 22.97) - open/close 14465 (time 1787123758.66 total 856.09 last 18.49) - open/close 14667 (time 1787123768.66 total 866.09 last 20.20) - open/close 14809 (time 1787123779.20 total 876.63 last 13.47) - open/close 14914 (time 1787123789.57 total 887.00 last 10.13) - open/close 15176 (time 1787123800.38 total 897.81 last 24.25) - open/close 15410 (time 1787123810.38 total 907.81 last 23.38) - open/close 15658 (time 1787123825.88 total 923.31 last 16.01) - open/close 15849 (time 1787123835.88 total 933.31 last 19.09) - open/close 16113 (time 1787123846.40 total 943.83 last 25.11) - open/close 16369 (time 1787123856.77 total 954.20 last 24.68) - open/close 16562 (time 1787123867.64 total 965.07 last 17.75) - open/close 16706 (time 1787123878.59 total 976.02 last 13.15) - open/close 16822 (time 1787123889.15 total 986.58 last 10.98) - open/close 16993 (time 1787123900.09 total 997.52 last 15.64) - open/close 17137 (time 1787123910.21 total 1007.64 last 14.23) - open/close 17317 (time 1787123920.21 total 1017.64 last 18.00) - open/close 17767 (time 1787123930.21 total 1027.64 last 44.98) - open/close 18310 (time 1787123940.22 total 1037.65 last 54.25) - open/close 18870 (time 1787123950.23 total 1047.66 last 55.99) - open/close 19438 (time 1787123960.24 total 1057.67 last 56.70) - open/close 20000 (time 1787123968.80 total 1066.23 last 65.66) - open/close 20630 (time 1787123978.81 total 1076.24 last 62.95) - open/close 21242 (time 1787123988.84 total 1086.27 last 61.04) - open/close 21758 (time 1787123998.87 total 1096.30 last 51.45) - open/close 22234 (time 1787124008.87 total 1106.30 last 47.58) - open/close 22835 (time 1787124018.87 total 1116.30 last 60.10) - open/close 23480 (time 1787124028.89 total 1126.32 last 64.40) - open/close 24103 (time 1787124038.90 total 1136.33 last 62.23) - open/close 24720 (time 1787124048.90 total 1146.33 last 61.66) - open/close 25301 (time 1787124058.92 total 1156.35 last 58.01) - open/close 25910 (time 1787124068.93 total 1166.36 last 60.82) - open/close 26539 (time 1787124078.93 total 1176.36 last 62.88) - open/close 27157 (time 1787124088.94 total 1186.37 last 61.78) - open/close 27722 (time 1787124098.95 total 1196.38 last 56.44) - open/close 28329 (time 1787124108.96 total 1206.39 last 60.64) - open/close 28943 (time 1787124118.97 total 1216.40 last 61.35) - open/close 29572 (time 1787124128.98 total 1226.41 last 62.84) - open/close 30000 (time 1787124135.11 total 1232.54 last 69.79) - open/close 30676 (time 1787124145.11 total 1242.54 last 67.57) - open/close 31343 (time 1787124155.12 total 1252.55 last 66.66) - open/close 31967 (time 1787124165.13 total 1262.56 last 62.35) - open/close 32623 (time 1787124175.13 total 1272.56 last 65.57) - open/close 33310 (time 1787124185.14 total 1282.57 last 68.67) - open/close 33976 (time 1787124195.15 total 1292.58 last 66.50) - open/close 34573 (time 1787124205.15 total 1302.58 last 59.70) - open/close 35181 (time 1787124215.17 total 1312.60 last 60.70) - open/close 35827 (time 1787124225.17 total 1322.60 last 64.56) - open/close 36466 (time 1787124235.18 total 1332.61 last 63.83) - open/close 37073 (time 1787124245.19 total 1342.62 last 60.64) - open/close 37670 (time 1787124255.21 total 1352.64 last 59.62) - open/close 38328 (time 1787124265.25 total 1362.68 last 65.55) - open/close 38976 (time 1787124275.25 total 1372.68 last 64.79) - open/close 39551 (time 1787124285.26 total 1382.69 last 57.46) - open/close 40000 (time 1787124292.96 total 1390.39 last 58.29) - open/close 40660 (time 1787124302.97 total 1400.40 last 65.92) - open/close 41278 (time 1787124312.97 total 1410.40 last 61.80) - open/close 41928 (time 1787124322.99 total 1420.42 last 64.91) - open/close 42501 (time 1787124332.99 total 1430.42 last 57.27) - open/close 43063 (time 1787124343.01 total 1440.44 last 56.11) - open/close 43646 (time 1787124353.02 total 1450.45 last 58.23) - open/close 44224 (time 1787124363.03 total 1460.46 last 57.73) - open/close 44917 (time 1787124373.04 total 1470.47 last 69.25) - open/close 45469 (time 1787124383.07 total 1480.50 last 55.06) - open/close 46144 (time 1787124393.09 total 1490.52 last 67.36) - open/close 46770 (time 1787124403.12 total 1500.55 last 62.41) - open/close 47474 (time 1787124413.14 total 1510.57 last 70.23) - open/close 48008 (time 1787124423.15 total 1520.58 last 53.37) - open/close 48571 (time 1787124433.15 total 1530.58 last 56.29) - open/close 49152 (time 1787124443.20 total 1540.63 last 57.80) - open/close 49687 (time 1787124453.20 total 1550.63 last 53.50) total: 50000 open/close in 1556.31 seconds: 32.13 ops/second - unlinked 0 (time 1787124463 ; total 0 ; last 0) - unlinked 10000 (time 1787124632 ; total 169 ; last 169) - unlinked 20000 (time 1787124813 ; total 350 ; last 181) - unlinked 30000 (time 1787124993 ; total 530 ; last 180) - unlinked 40000 (time 1787125163 ; total 700 ; last 170) total: 50000 unlinks in 872 seconds: 57.339451 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: modules unloaded. pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2