== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:13:34 (1763338414) start mds service on oleg217-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing load_modules_local oleg217-server: Loading modules from /home/green/git/lustre-release/lustre oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg217-server: detected 4 online CPUs by sysfs oleg217-server: MODOPTS_LIBCFS= oleg217-server: Force libcfs to create 2 CPU partitions oleg217-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg217-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg217-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg217-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg217-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg217-server: oleg217-server.virtnet: executing set_default_debug -1 all pdsh@oleg217-client: oleg217-server: ssh exited with exit code 1 Started lustre-OST0000 oleg217-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg217-client.virtnet: -o user_xattr,flock 192.168.202.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1763338508.37 total 10.04 last 18.02) - open/close 390 (time 1763338518.73 total 20.40 last 20.18) - open/close 673 (time 1763338529.61 total 31.28 last 26.00) - open/close 891 (time 1763338539.63 total 41.31 last 21.75) - open/close 1107 (time 1763338549.64 total 51.32 last 21.57) - open/close 1276 (time 1763338560.53 total 62.21 last 15.52) - open/close 1444 (time 1763338570.64 total 72.32 last 16.62) - open/close 1616 (time 1763338580.64 total 82.32 last 17.20) - open/close 1844 (time 1763338590.66 total 92.33 last 22.77) - open/close 2013 (time 1763338600.66 total 102.34 last 16.89) - open/close 2222 (time 1763338611.34 total 113.02 last 19.57) - open/close 2427 (time 1763338621.45 total 123.12 last 20.29) - open/close 2713 (time 1763338631.78 total 133.45 last 27.69) - open/close 2998 (time 1763338642.77 total 144.44 last 25.93) - open/close 3215 (time 1763338653.76 total 155.44 last 19.74) - open/close 3409 (time 1763338664.11 total 165.79 last 18.75) - open/close 3626 (time 1763338675.02 total 176.70 last 19.89) - open/close 3844 (time 1763338685.04 total 186.72 last 21.75) - open/close 4101 (time 1763338696.07 total 197.74 last 23.32) - open/close 4347 (time 1763338706.69 total 208.36 last 23.16) - open/close 4528 (time 1763338716.70 total 218.38 last 18.07) - open/close 4748 (time 1763338726.71 total 228.38 last 22.00) - open/close 4899 (time 1763338736.75 total 238.42 last 15.04) - open/close 5144 (time 1763338746.83 total 248.50 last 24.30) - open/close 5374 (time 1763338757.45 total 259.12 last 21.66) - open/close 5560 (time 1763338767.62 total 269.30 last 18.28) - open/close 5757 (time 1763338777.86 total 279.54 last 19.24) - open/close 5917 (time 1763338788.36 total 290.04 last 15.24) - open/close 6128 (time 1763338799.00 total 300.68 last 19.83) - open/close 6333 (time 1763338809.55 total 311.22 last 19.45) - open/close 6487 (time 1763338820.30 total 321.98 last 14.32) - open/close 6672 (time 1763338830.68 total 332.36 last 17.82) - open/close 6899 (time 1763338840.69 total 342.37 last 22.68) - open/close 7132 (time 1763338850.69 total 352.37 last 23.30) - open/close 7347 (time 1763338861.34 total 363.02 last 20.19) - open/close 7566 (time 1763338872.01 total 373.68 last 20.53) - open/close 7815 (time 1763338882.39 total 384.06 last 23.98) - open/close 8058 (time 1763338892.83 total 394.51 last 23.27) - open/close 8278 (time 1763338903.05 total 404.73 last 21.53) - open/close 8516 (time 1763338913.54 total 415.22 last 22.69) - open/close 8749 (time 1763338924.43 total 426.11 last 21.39) - open/close 8978 (time 1763338934.62 total 436.29 last 22.49) - open/close 9210 (time 1763338945.04 total 446.71 last 22.26) - open/close 9498 (time 1763338955.73 total 457.41 last 26.93) - open/close 9723 (time 1763338966.16 total 467.83 last 21.58) - open/close 9942 (time 1763338976.40 total 478.08 last 21.37) - open/close 10000 (time 1763338978.64 total 480.32 last 25.90) - open/close 10150 (time 1763338988.82 total 490.49 last 14.75) - open/close 10355 (time 1763338999.12 total 500.80 last 19.89) - open/close 10548 (time 1763339009.29 total 510.97 last 18.98) - open/close 10622 (time 1763339020.15 total 521.83 last 6.81) - open/close 10787 (time 1763339031.63 total 533.30 last 14.38) - open/close 10983 (time 1763339041.94 total 543.62 last 19.00) - open/close 11242 (time 1763339052.42 total 554.10 last 24.72) - open/close 11428 (time 1763339062.79 total 564.47 last 17.93) - open/close 11659 (time 1763339072.80 total 574.47 last 23.09) - open/close 11892 (time 1763339083.80 total 585.47 last 21.19) - open/close 12157 (time 1763339094.41 total 596.09 last 24.96) - open/close 12399 (time 1763339104.67 total 606.35 last 23.59) - open/close 12608 (time 1763339114.97 total 616.65 last 20.28) - open/close 12818 (time 1763339125.64 total 627.31 last 19.69) - open/close 13015 (time 1763339135.96 total 637.64 last 19.08) - open/close 13250 (time 1763339146.19 total 647.87 last 22.97) - open/close 13422 (time 1763339156.20 total 657.88 last 17.18) - open/close 13619 (time 1763339166.23 total 667.90 last 19.65) - open/close 13804 (time 1763339179.36 total 681.04 last 14.09) - open/close 14070 (time 1763339189.36 total 691.04 last 26.59) - open/close 14270 (time 1763339199.37 total 701.05 last 19.98) - open/close 14456 (time 1763339209.38 total 711.06 last 18.59) - open/close 14678 (time 1763339219.40 total 721.08 last 22.15) - open/close 14846 (time 1763339229.61 total 731.29 last 16.46) - open/close 15112 (time 1763339239.64 total 741.31 last 26.53) - open/close 15384 (time 1763339250.63 total 752.31 last 24.74) - open/close 15647 (time 1763339261.57 total 763.25 last 24.04) - open/close 15890 (time 1763339272.41 total 774.09 last 22.42) - open/close 16142 (time 1763339282.83 total 784.51 last 24.20) - open/close 16319 (time 1763339295.24 total 796.92 last 14.26) - open/close 16541 (time 1763339305.83 total 807.51 last 20.96) - open/close 16699 (time 1763339316.05 total 817.73 last 15.47) - open/close 17440 (time 1763339326.06 total 827.74 last 73.99) - open/close 18209 (time 1763339336.07 total 837.75 last 76.86) - open/close 18911 (time 1763339346.07 total 847.75 last 70.18) - open/close 19702 (time 1763339356.08 total 857.75 last 79.08) - open/close 20000 (time 1763339359.82 total 861.50 last 79.51) - open/close 20783 (time 1763339369.83 total 871.51 last 78.23) - open/close 21597 (time 1763339379.85 total 881.52 last 81.30) - open/close 22321 (time 1763339389.85 total 891.52 last 72.38) - open/close 23079 (time 1763339399.85 total 901.53 last 75.79) - open/close 23850 (time 1763339409.86 total 911.53 last 77.05) - open/close 24632 (time 1763339419.87 total 921.55 last 78.10) - open/close 25400 (time 1763339429.87 total 931.55 last 76.78) - open/close 26196 (time 1763339439.89 total 941.57 last 79.44) - open/close 26979 (time 1763339449.91 total 951.58 last 78.18) - open/close 27786 (time 1763339459.92 total 961.60 last 80.59) - open/close 28553 (time 1763339469.94 total 971.61 last 76.57) - open/close 29299 (time 1763339479.94 total 981.62 last 74.57) - open/close 29987 (time 1763339489.95 total 991.63 last 68.76) - open/close 30701 (time 1763339499.96 total 1001.63 last 71.34) - open/close 31507 (time 1763339509.97 total 1011.64 last 80.53) - open/close 32347 (time 1763339519.97 total 1021.65 last 83.96) - open/close 33165 (time 1763339529.98 total 1031.65 last 81.75) - open/close 34053 (time 1763339539.99 total 1041.66 last 88.73) - open/close 34911 (time 1763339549.99 total 1051.67 last 85.75) - open/close 35711 (time 1763339560.01 total 1061.68 last 79.87) - open/close 36617 (time 1763339570.01 total 1071.69 last 90.55) - open/close 37445 (time 1763339580.01 total 1081.69 last 82.77) - open/close 38334 (time 1763339590.03 total 1091.70 last 88.80) - open/close 39189 (time 1763339600.03 total 1101.70 last 85.50) - open/close 39967 (time 1763339610.04 total 1111.71 last 77.72) - open/close 40805 (time 1763339620.04 total 1121.72 last 83.78) - open/close 41638 (time 1763339630.04 total 1131.72 last 83.29) - open/close 42494 (time 1763339640.04 total 1141.72 last 85.59) - open/close 43305 (time 1763339650.04 total 1151.72 last 81.09) - open/close 44113 (time 1763339660.07 total 1161.74 last 80.62) - open/close 44913 (time 1763339670.07 total 1171.74 last 80.00) - open/close 45700 (time 1763339680.08 total 1181.76 last 78.58) - open/close 46485 (time 1763339690.09 total 1191.76 last 78.45) - open/close 47289 (time 1763339700.09 total 1201.77 last 80.38) - open/close 48107 (time 1763339710.10 total 1211.78 last 81.72) - open/close 48916 (time 1763339720.10 total 1221.78 last 80.89) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1230.55 seconds: 40.33 ops/second - unlinked 0 (time 1763339732 ; total 0 ; last 0) - unlinked 10000 (time 1763339859 ; total 127 ; last 127) - unlinked 20000 (time 1763339993 ; total 261 ; last 134) - unlinked 30000 (time 1763340110 ; total 378 ; last 117) - unlinked 40000 (time 1763340233 ; total 501 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 621 seconds: 79.922707 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg217-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg217-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg217-server stop mds service on oleg217-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg217-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg217-server' oleg217-server: oleg217-server.virtnet: executing unload_modules_local oleg217-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg217-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-client: ssh exited with exit code 2 pdsh@oleg217-client: oleg217-server: ssh exited with exit code 2