== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:17:20 (1766416640) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg445-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 198 (time 1766416712.53 total 10.00 last 19.79) - open/close 353 (time 1766416722.94 total 20.42 last 14.89) - open/close 544 (time 1766416732.95 total 30.42 last 19.10) - open/close 798 (time 1766416743.70 total 41.17 last 23.62) - open/close 958 (time 1766416754.17 total 51.65 last 15.28) - open/close 1130 (time 1766416764.93 total 62.40 last 15.99) - open/close 1379 (time 1766416775.57 total 73.05 last 23.39) - open/close 1564 (time 1766416785.58 total 83.05 last 18.49) - open/close 1750 (time 1766416795.60 total 93.08 last 18.56) - open/close 1959 (time 1766416805.69 total 103.17 last 20.71) - open/close 2197 (time 1766416816.45 total 113.92 last 22.13) - open/close 2395 (time 1766416826.50 total 123.98 last 19.69) - open/close 2520 (time 1766416836.51 total 133.99 last 12.49) - open/close 2712 (time 1766416846.52 total 143.99 last 19.19) - open/close 2920 (time 1766416857.53 total 155.01 last 18.89) - open/close 3060 (time 1766416867.84 total 165.31 last 13.59) - open/close 3167 (time 1766416878.48 total 175.95 last 10.06) - open/close 3316 (time 1766416889.47 total 186.94 last 13.56) - open/close 3390 (time 1766416899.78 total 197.25 last 7.18) - open/close 3551 (time 1766416909.96 total 207.43 last 15.81) - open/close 3705 (time 1766416921.03 total 218.50 last 13.91) - open/close 3976 (time 1766416931.97 total 229.44 last 24.78) - open/close 4152 (time 1766416942.90 total 240.38 last 16.09) - open/close 4426 (time 1766416953.74 total 251.21 last 25.29) - open/close 4743 (time 1766416963.74 total 261.21 last 31.70) - open/close 5037 (time 1766416974.52 total 271.99 last 27.27) - open/close 5262 (time 1766416985.09 total 282.56 last 21.30) - open/close 5516 (time 1766416998.02 total 295.49 last 19.64) - open/close 5734 (time 1766417008.26 total 305.74 last 21.29) - open/close 5953 (time 1766417018.44 total 315.92 last 21.51) - open/close 6294 (time 1766417028.44 total 325.92 last 34.09) - open/close 6586 (time 1766417038.45 total 335.93 last 29.18) - open/close 6900 (time 1766417048.45 total 345.93 last 31.39) - open/close 7111 (time 1766417059.33 total 356.80 last 19.40) - open/close 7357 (time 1766417069.82 total 367.29 last 23.45) - open/close 7662 (time 1766417080.57 total 378.04 last 28.37) - open/close 7948 (time 1766417091.01 total 388.48 last 27.39) - open/close 8173 (time 1766417101.88 total 399.35 last 20.70) - open/close 8482 (time 1766417111.89 total 409.37 last 30.86) - open/close 8724 (time 1766417122.37 total 419.84 last 23.11) - open/close 8999 (time 1766417133.05 total 430.53 last 25.74) - open/close 9321 (time 1766417143.86 total 441.34 last 29.78) - open/close 9557 (time 1766417154.11 total 451.59 last 23.03) - open/close 9884 (time 1766417164.86 total 462.34 last 30.42) - open/close 10000 (time 1766417167.75 total 465.22 last 40.22) - open/close 10262 (time 1766417177.85 total 475.32 last 25.93) - open/close 10592 (time 1766417187.86 total 485.33 last 32.98) - open/close 10833 (time 1766417198.79 total 496.26 last 22.05) - open/close 11111 (time 1766417209.15 total 506.62 last 26.83) - open/close 11433 (time 1766417219.77 total 517.25 last 30.30) - open/close 11854 (time 1766417229.78 total 527.25 last 42.08) - open/close 13316 (time 1766417239.78 total 537.25 last 146.20) - open/close 14868 (time 1766417249.78 total 547.26 last 155.16) - open/close 16365 (time 1766417259.79 total 557.26 last 149.63) - open/close 17927 (time 1766417269.80 total 567.27 last 156.01) - open/close 19519 (time 1766417279.80 total 577.27 last 159.16) - open/close 20000 (time 1766417283.24 total 580.71 last 139.92) - open/close 21531 (time 1766417293.24 total 590.71 last 153.09) - open/close 22902 (time 1766417303.24 total 600.71 last 137.07) - open/close 24561 (time 1766417313.24 total 610.72 last 165.85) - open/close 26402 (time 1766417323.24 total 620.72 last 184.08) - open/close 28125 (time 1766417333.25 total 630.72 last 172.23) - open/close 29803 (time 1766417343.25 total 640.73 last 167.70) - open/close 31364 (time 1766417353.26 total 650.73 last 156.06) - open/close 32822 (time 1766417363.26 total 660.73 last 145.76) - open/close 34553 (time 1766417373.26 total 670.74 last 173.07) - open/close 36330 (time 1766417383.26 total 680.74 last 177.69) - open/close 38003 (time 1766417393.27 total 690.74 last 167.22) - open/close 39514 (time 1766417403.27 total 700.75 last 151.03) - open/close 40000 (time 1766417406.22 total 703.70 last 164.72) - open/close 41528 (time 1766417416.22 total 713.70 last 152.79) - open/close 43056 (time 1766417426.23 total 723.70 last 152.73) - open/close 44638 (time 1766417436.23 total 733.70 last 158.19) - open/close 46180 (time 1766417446.24 total 743.72 last 153.95) - open/close 47691 (time 1766417456.25 total 753.73 last 150.99) - open/close 49340 (time 1766417466.26 total 763.73 last 164.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 765.49 seconds: 64.84 ops/second - unlinked 0 (time 1766417469 ; total 0 ; last 0) - unlinked 10000 (time 1766417514 ; total 45 ; last 45) - unlinked 20000 (time 1766417548 ; total 79 ; last 34) - unlinked 30000 (time 1766417584 ; total 115 ; last 36) - unlinked 40000 (time 1766417618 ; total 149 ; last 34) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 180 seconds: 275.733337 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg445-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2