== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:34:14 (1763336054) start mds service on oleg158-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 195 (time 1763336136.68 total 10.04 last 19.42) - open/close 388 (time 1763336146.70 total 20.06 last 19.26) - open/close 600 (time 1763336157.23 total 30.59 last 20.15) - open/close 821 (time 1763336167.60 total 40.96 last 21.31) - open/close 1029 (time 1763336177.61 total 50.97 last 20.77) - open/close 1231 (time 1763336188.67 total 62.03 last 18.27) - open/close 1449 (time 1763336198.95 total 72.31 last 21.19) - open/close 1705 (time 1763336208.97 total 82.33 last 25.56) - open/close 1859 (time 1763336219.36 total 92.72 last 14.82) - open/close 2000 (time 1763336229.37 total 102.73 last 14.10) - open/close 2137 (time 1763336240.17 total 113.53 last 12.68) - open/close 2398 (time 1763336250.18 total 123.54 last 26.07) - open/close 2581 (time 1763336261.23 total 134.59 last 16.55) - open/close 2824 (time 1763336271.24 total 144.60 last 24.29) - open/close 3034 (time 1763336282.02 total 155.38 last 19.48) - open/close 3239 (time 1763336292.52 total 165.88 last 19.52) - open/close 3420 (time 1763336303.02 total 176.38 last 17.24) - open/close 3620 (time 1763336313.06 total 186.42 last 19.92) - open/close 3838 (time 1763336323.12 total 196.48 last 21.69) - open/close 4013 (time 1763336336.11 total 209.47 last 13.46) - open/close 4243 (time 1763336346.22 total 219.58 last 22.77) - open/close 4482 (time 1763336357.11 total 230.47 last 21.94) - open/close 4701 (time 1763336367.98 total 241.34 last 20.15) - open/close 4899 (time 1763336378.81 total 252.17 last 18.28) - open/close 5141 (time 1763336388.83 total 262.19 last 24.13) - open/close 5431 (time 1763336398.85 total 272.21 last 28.97) - open/close 5669 (time 1763336409.00 total 282.36 last 23.44) - open/close 5845 (time 1763336419.76 total 293.12 last 16.36) - open/close 6057 (time 1763336430.06 total 303.42 last 20.58) - open/close 6279 (time 1763336440.06 total 313.42 last 22.18) - open/close 6467 (time 1763336450.16 total 323.52 last 18.62) - open/close 6696 (time 1763336460.34 total 333.70 last 22.49) - open/close 6868 (time 1763336471.27 total 344.63 last 15.74) - open/close 7037 (time 1763336481.90 total 355.26 last 15.90) - open/close 7190 (time 1763336491.92 total 365.28 last 15.26) - open/close 7376 (time 1763336502.31 total 375.67 last 17.91) - open/close 7639 (time 1763336513.25 total 386.61 last 24.03) - open/close 7790 (time 1763336523.56 total 396.92 last 14.65) - open/close 7912 (time 1763336533.67 total 407.03 last 12.07) - open/close 8059 (time 1763336544.11 total 417.47 last 14.08) - open/close 8275 (time 1763336554.34 total 427.70 last 21.11) - open/close 8434 (time 1763336564.78 total 438.14 last 15.24) - open/close 8567 (time 1763336574.78 total 448.14 last 13.30) - open/close 8691 (time 1763336585.83 total 459.19 last 11.22) - open/close 8889 (time 1763336595.84 total 469.20 last 19.78) - open/close 9088 (time 1763336606.72 total 480.08 last 18.30) - open/close 9337 (time 1763336616.72 total 490.08 last 24.89) - open/close 9545 (time 1763336626.86 total 500.22 last 20.52) - open/close 9741 (time 1763336637.74 total 511.10 last 18.02) - open/close 9918 (time 1763336648.43 total 521.79 last 16.56) - open/close 10000 (time 1763336655.68 total 529.04 last 11.31) - open/close 10146 (time 1763336665.83 total 539.19 last 14.38) - open/close 10320 (time 1763336676.39 total 549.75 last 16.47) - open/close 10583 (time 1763336687.27 total 560.64 last 24.17) - open/close 10799 (time 1763336697.40 total 570.76 last 21.34) - open/close 11002 (time 1763336707.40 total 580.76 last 20.29) - open/close 11117 (time 1763336718.19 total 591.55 last 10.66) - open/close 11235 (time 1763336728.82 total 602.18 last 11.10) - open/close 11285 (time 1763336738.92 total 612.28 last 4.95) - open/close 11404 (time 1763336748.92 total 622.28 last 11.89) - open/close 11575 (time 1763336759.53 total 632.89 last 16.12) - open/close 11810 (time 1763336770.16 total 643.52 last 22.12) - open/close 12014 (time 1763336780.45 total 653.81 last 19.81) - open/close 12271 (time 1763336791.46 total 664.82 last 23.35) - open/close 12511 (time 1763336802.09 total 675.45 last 22.58) - open/close 12775 (time 1763336812.79 total 686.15 last 24.68) - open/close 13320 (time 1763336822.80 total 696.16 last 54.45) - open/close 14217 (time 1763336832.81 total 706.17 last 89.60) - open/close 15139 (time 1763336842.81 total 716.17 last 92.20) - open/close 16186 (time 1763336852.81 total 726.17 last 104.67) - open/close 17360 (time 1763336862.83 total 736.19 last 117.25) - open/close 18451 (time 1763336872.83 total 746.19 last 109.03) - open/close 19463 (time 1763336882.84 total 756.20 last 101.13) - open/close 20000 (time 1763336888.08 total 761.44 last 102.52) - open/close 21011 (time 1763336898.08 total 771.44 last 101.08) - open/close 21968 (time 1763336908.09 total 781.45 last 95.62) - open/close 22875 (time 1763336918.10 total 791.46 last 90.59) - open/close 23643 (time 1763336928.10 total 801.46 last 76.76) - open/close 24399 (time 1763336938.11 total 811.47 last 75.55) - open/close 25254 (time 1763336948.11 total 821.47 last 85.50) - open/close 26206 (time 1763336958.15 total 831.51 last 94.81) - open/close 27083 (time 1763336968.16 total 841.52 last 87.59) - open/close 27994 (time 1763336978.16 total 851.52 last 91.10) - open/close 28874 (time 1763336988.17 total 861.53 last 87.96) - open/close 29781 (time 1763336998.18 total 871.54 last 90.63) - open/close 30000 (time 1763337000.82 total 874.18 last 82.88) - open/close 30831 (time 1763337010.82 total 884.19 last 83.06) - open/close 31652 (time 1763337020.83 total 894.19 last 82.06) - open/close 32507 (time 1763337030.84 total 904.20 last 85.43) - open/close 33420 (time 1763337040.85 total 914.21 last 91.21) - open/close 34291 (time 1763337050.86 total 924.22 last 87.04) - open/close 35054 (time 1763337060.86 total 934.22 last 76.26) - open/close 35910 (time 1763337070.87 total 944.23 last 85.50) - open/close 36836 (time 1763337080.87 total 954.23 last 92.58) - open/close 37746 (time 1763337090.88 total 964.24 last 90.95) - open/close 38669 (time 1763337100.88 total 974.24 last 92.26) - open/close 39571 (time 1763337110.90 total 984.26 last 90.06) - open/close 40000 (time 1763337115.93 total 989.29 last 85.32) - open/close 40782 (time 1763337125.94 total 999.30 last 78.10) - open/close 41683 (time 1763337135.95 total 1009.31 last 90.02) - open/close 42581 (time 1763337145.95 total 1019.31 last 89.76) - open/close 43467 (time 1763337155.95 total 1029.32 last 88.58) - open/close 44325 (time 1763337165.96 total 1039.32 last 85.78) - open/close 45131 (time 1763337175.96 total 1049.32 last 80.57) - open/close 45989 (time 1763337185.97 total 1059.33 last 85.72) - open/close 46907 (time 1763337195.97 total 1069.33 last 91.79) - open/close 47798 (time 1763337205.98 total 1079.34 last 89.06) - open/close 48624 (time 1763337215.99 total 1089.35 last 82.48) - open/close 49456 (time 1763337226.00 total 1099.36 last 83.13) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1101.23 seconds: 45.07 ops/second - unlinked 0 (time 1763337231 ; total 0 ; last 0) - unlinked 10000 (time 1763337348 ; total 117 ; last 117) - unlinked 20000 (time 1763337425 ; total 194 ; last 77) - unlinked 30000 (time 1763337492 ; total 261 ; last 67) - unlinked 40000 (time 1763337557 ; total 326 ; last 65) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 390 seconds: 127.261536 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2