== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:51:53 (1763344313) start mds service on oleg159-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 252 (time 1763344413.70 total 10.90 last 23.11) - open/close 509 (time 1763344424.64 total 21.84 last 23.49) - open/close 722 (time 1763344435.16 total 32.36 last 20.26) - open/close 875 (time 1763344445.58 total 42.78 last 14.68) - open/close 1087 (time 1763344455.70 total 52.90 last 20.95) - open/close 1204 (time 1763344467.89 total 65.09 last 9.60) - open/close 1467 (time 1763344478.93 total 76.13 last 23.82) - open/close 1652 (time 1763344488.93 total 86.13 last 18.49) - open/close 1870 (time 1763344498.95 total 96.15 last 21.77) - open/close 2084 (time 1763344508.95 total 106.15 last 21.38) - open/close 2283 (time 1763344519.44 total 116.64 last 18.98) - open/close 2502 (time 1763344529.66 total 126.86 last 21.42) - open/close 2716 (time 1763344540.04 total 137.24 last 20.62) - open/close 2985 (time 1763344550.06 total 147.26 last 26.84) - open/close 3191 (time 1763344560.08 total 157.28 last 20.58) - open/close 3378 (time 1763344570.56 total 167.76 last 17.83) - open/close 3653 (time 1763344581.51 total 178.71 last 25.13) - open/close 3898 (time 1763344592.49 total 189.69 last 22.30) - open/close 4105 (time 1763344602.51 total 199.71 last 20.67) - open/close 4324 (time 1763344612.52 total 209.72 last 21.88) - open/close 4519 (time 1763344622.66 total 219.86 last 19.23) - open/close 4694 (time 1763344632.66 total 229.86 last 17.49) - open/close 4917 (time 1763344642.68 total 239.88 last 22.27) - open/close 5086 (time 1763344652.69 total 249.89 last 16.88) - open/close 5316 (time 1763344662.92 total 260.12 last 22.48) - open/close 5540 (time 1763344673.50 total 270.70 last 21.16) - open/close 5759 (time 1763344683.51 total 280.71 last 21.88) - open/close 5898 (time 1763344693.52 total 290.72 last 13.89) - open/close 6074 (time 1763344706.18 total 303.38 last 13.90) - open/close 6305 (time 1763344716.57 total 313.77 last 22.23) - open/close 6561 (time 1763344726.58 total 323.78 last 25.59) - open/close 6808 (time 1763344737.22 total 334.42 last 23.20) - open/close 7029 (time 1763344747.46 total 344.66 last 21.59) - open/close 7256 (time 1763344757.47 total 354.67 last 22.69) - open/close 7464 (time 1763344768.27 total 365.47 last 19.26) - open/close 7683 (time 1763344778.27 total 375.47 last 21.89) - open/close 7894 (time 1763344789.06 total 386.26 last 19.56) - open/close 8078 (time 1763344799.07 total 396.27 last 18.38) - open/close 8315 (time 1763344809.34 total 406.55 last 23.06) - open/close 8531 (time 1763344819.85 total 417.05 last 20.57) - open/close 8781 (time 1763344829.86 total 427.07 last 24.95) - open/close 8971 (time 1763344839.88 total 437.08 last 18.98) - open/close 9191 (time 1763344849.88 total 447.08 last 21.99) - open/close 9365 (time 1763344859.89 total 457.09 last 17.39) - open/close 9583 (time 1763344869.90 total 467.10 last 21.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9746) error: No space left on device total: 9746 open/close in 474.83 seconds: 20.53 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8872: kill: (111799) - No such process - unlinked 0 (time 1763344885 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9746) error: No such file or directory total: 9746 unlinks in 69 seconds: 141.246384 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2