== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:34:37 (1785224077) start mds service on oleg231-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg231-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg231-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg231-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 166 (time 1785224171.79 total 10.44 last 15.89) - open/close 499 (time 1785224182.76 total 21.41 last 30.36) - open/close 776 (time 1785224192.99 total 31.65 last 27.07) - open/close 1043 (time 1785224203.01 total 41.66 last 26.66) - open/close 1258 (time 1785224213.74 total 52.39 last 20.04) - open/close 1542 (time 1785224224.02 total 62.67 last 27.62) - open/close 1845 (time 1785224234.18 total 72.84 last 29.81) - open/close 2048 (time 1785224244.42 total 83.07 last 19.83) - open/close 2294 (time 1785224255.24 total 93.90 last 22.73) - open/close 2482 (time 1785224265.26 total 103.91 last 18.77) - open/close 2717 (time 1785224276.16 total 114.82 last 21.55) - open/close 2920 (time 1785224286.37 total 125.02 last 19.90) - open/close 3111 (time 1785224296.91 total 135.56 last 18.12) - open/close 3407 (time 1785224307.91 total 146.57 last 26.90) - open/close 3601 (time 1785224317.91 total 156.57 last 19.40) - open/close 3862 (time 1785224328.95 total 167.60 last 23.65) - open/close 4161 (time 1785224339.31 total 177.96 last 28.86) - open/close 4427 (time 1785224349.32 total 187.98 last 26.55) - open/close 4658 (time 1785224359.85 total 198.50 last 21.95) - open/close 4924 (time 1785224369.85 total 208.51 last 26.60) - open/close 5201 (time 1785224379.86 total 218.51 last 27.68) - open/close 5411 (time 1785224389.86 total 228.52 last 20.99) - open/close 5611 (time 1785224399.87 total 238.52 last 19.99) - open/close 5810 (time 1785224409.87 total 248.53 last 19.89) - open/close 6005 (time 1785224420.07 total 258.72 last 19.13) - open/close 6205 (time 1785224430.85 total 269.51 last 18.54) - open/close 6390 (time 1785224440.93 total 279.59 last 18.36) - open/close 6616 (time 1785224450.93 total 289.59 last 22.60) - open/close 6827 (time 1785224460.94 total 299.60 last 21.08) - open/close 7036 (time 1785224470.95 total 309.60 last 20.89) - open/close 7248 (time 1785224480.96 total 319.61 last 21.18) - open/close 7431 (time 1785224491.66 total 330.31 last 17.10) - open/close 7627 (time 1785224501.66 total 340.32 last 19.59) - open/close 7861 (time 1785224511.67 total 350.32 last 23.39) - open/close 7948 (time 1785224522.08 total 360.73 last 8.36) - open/close 8106 (time 1785224532.18 total 370.83 last 15.64) - open/close 8256 (time 1785224542.27 total 380.93 last 14.86) - open/close 8366 (time 1785224552.97 total 391.62 last 10.28) - open/close 8588 (time 1785224563.98 total 402.63 last 20.17) - open/close 8748 (time 1785224573.98 total 412.64 last 15.99) - open/close 8973 (time 1785224584.21 total 422.86 last 22.01) - open/close 9174 (time 1785224595.08 total 433.73 last 18.49) - open/close 9376 (time 1785224605.90 total 444.56 last 18.65) - open/close 9602 (time 1785224616.20 total 454.85 last 21.96) - open/close 9807 (time 1785224626.21 total 464.86 last 20.47) - open/close 10000 (time 1785224633.66 total 472.32 last 25.89) - open/close 10214 (time 1785224643.78 total 482.43 last 21.16) - open/close 10458 (time 1785224654.44 total 493.10 last 22.88) - open/close 10661 (time 1785224664.45 total 503.10 last 20.29) - open/close 10866 (time 1785224674.45 total 513.11 last 20.48) - open/close 11062 (time 1785224684.46 total 523.11 last 19.59) - open/close 11199 (time 1785224694.60 total 533.25 last 13.52) - open/close 11405 (time 1785224705.35 total 544.01 last 19.15) - open/close 11619 (time 1785224716.43 total 555.08 last 19.32) - open/close 11815 (time 1785224727.18 total 565.83 last 18.23) - open/close 12023 (time 1785224737.29 total 575.95 last 20.56) - open/close 12224 (time 1785224747.31 total 585.97 last 20.06) - open/close 12459 (time 1785224758.15 total 596.80 last 21.68) - open/close 12696 (time 1785224768.16 total 606.81 last 23.68) - open/close 12957 (time 1785224779.09 total 617.75 last 23.88) - open/close 13245 (time 1785224789.92 total 628.58 last 26.59) - open/close 13437 (time 1785224799.92 total 638.58 last 19.20) - open/close 13662 (time 1785224810.44 total 649.10 last 21.39) - open/close 13889 (time 1785224820.46 total 659.11 last 22.67) - open/close 14178 (time 1785224830.96 total 669.62 last 27.51) - open/close 14401 (time 1785224841.11 total 679.77 last 21.97) - open/close 14634 (time 1785224851.28 total 689.94 last 22.91) - open/close 14859 (time 1785224861.46 total 700.12 last 22.11) - open/close 15108 (time 1785224871.51 total 710.16 last 24.79) - open/close 15331 (time 1785224881.94 total 720.59 last 21.38) - open/close 15910 (time 1785224891.94 total 730.59 last 57.90) - open/close 16652 (time 1785224901.95 total 740.61 last 74.07) - open/close 17056 (time 1785224911.97 total 750.63 last 40.33) - open/close 17488 (time 1785224921.99 total 760.64 last 43.13) - open/close 17924 (time 1785224932.00 total 770.65 last 43.55) - open/close 18432 (time 1785224942.02 total 780.67 last 50.72) - open/close 19122 (time 1785224952.02 total 790.68 last 68.95) - open/close 19864 (time 1785224962.04 total 800.69 last 74.08) - open/close 20000 (time 1785224964.19 total 802.84 last 63.36) - open/close 20809 (time 1785224974.19 total 812.85 last 80.84) - open/close 21624 (time 1785224984.20 total 822.85 last 81.48) - open/close 22488 (time 1785224994.21 total 832.86 last 86.28) - open/close 23327 (time 1785225004.21 total 842.87 last 83.88) - open/close 24095 (time 1785225014.22 total 852.88 last 76.73) - open/close 24950 (time 1785225024.23 total 862.89 last 85.41) - open/close 25742 (time 1785225034.25 total 872.90 last 79.09) - open/close 26721 (time 1785225044.26 total 882.91 last 97.79) - open/close 27690 (time 1785225054.28 total 892.94 last 96.64) - open/close 28603 (time 1785225064.29 total 902.94 last 91.25) - open/close 29536 (time 1785225074.30 total 912.96 last 93.17) - open/close 30000 (time 1785225079.51 total 918.16 last 89.15) - open/close 30887 (time 1785225089.52 total 928.17 last 88.62) - open/close 31758 (time 1785225099.52 total 938.18 last 87.05) - open/close 32741 (time 1785225109.54 total 948.19 last 98.13) - open/close 33602 (time 1785225119.55 total 958.20 last 86.01) - open/close 34530 (time 1785225129.55 total 968.21 last 92.79) - open/close 35219 (time 1785225139.55 total 978.21 last 68.87) - open/close 35988 (time 1785225149.57 total 988.23 last 76.76) - open/close 36835 (time 1785225159.58 total 998.23 last 84.68) - open/close 37694 (time 1785225169.59 total 1008.24 last 85.82) - open/close 38494 (time 1785225179.59 total 1018.24 last 79.99) - open/close 39296 (time 1785225189.60 total 1028.26 last 80.08) - open/close 39967 (time 1785225199.60 total 1038.26 last 67.09) - open/close 40800 (time 1785225209.61 total 1048.27 last 83.22) - open/close 41718 (time 1785225219.62 total 1058.27 last 91.77) - open/close 42492 (time 1785225229.64 total 1068.29 last 77.26) - open/close 43151 (time 1785225239.65 total 1078.30 last 65.83) - open/close 43918 (time 1785225249.65 total 1088.31 last 76.65) - open/close 44797 (time 1785225259.66 total 1098.31 last 87.84) - open/close 45742 (time 1785225269.67 total 1108.32 last 94.42) - open/close 46558 (time 1785225279.67 total 1118.32 last 81.59) - open/close 47434 (time 1785225289.67 total 1128.33 last 87.55) - open/close 48325 (time 1785225299.68 total 1138.34 last 89.05) - open/close 49238 (time 1785225309.68 total 1148.34 last 91.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1152.97 seconds: 43.05 ops/second - unlinked 0 (time 1785225319 ; total 0 ; last 0) - unlinked 10000 (time 1785225444 ; total 125 ; last 125) - unlinked 20000 (time 1785225567 ; total 248 ; last 123) - unlinked 30000 (time 1785225648 ; total 329 ; last 81) - unlinked 40000 (time 1785225717 ; total 398 ; last 69) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 458 seconds: 108.364632 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg231-server unloading modules via unload_modules_local on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2