== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:37:07 (1768142227) start mds service on oleg438-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing load_modules_local oleg438-server: Loading modules from /home/green/git/lustre-release/lustre oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg438-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg438-server: detected 4 online CPUs by sysfs oleg438-server: MODOPTS_LIBCFS= oleg438-server: Force libcfs to create 2 CPU partitions oleg438-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg438-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg438-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg438-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg438-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug -1 all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 517 (time 1768142286.25 total 10.10 last 51.20) - open/close 1000 (time 1768142296.68 total 20.53 last 46.29) - open/close 1532 (time 1768142307.30 total 31.15 last 50.09) - open/close 2022 (time 1768142317.61 total 41.46 last 47.55) - open/close 2570 (time 1768142328.29 total 52.15 last 51.27) - open/close 3062 (time 1768142338.30 total 62.15 last 49.19) - open/close 3571 (time 1768142348.77 total 72.63 last 48.58) - open/close 4178 (time 1768142359.53 total 83.38 last 56.45) - open/close 4751 (time 1768142370.28 total 94.14 last 53.27) - open/close 5266 (time 1768142380.84 total 104.69 last 48.79) - open/close 5770 (time 1768142391.27 total 115.12 last 48.31) - open/close 6261 (time 1768142401.70 total 125.56 last 47.07) - open/close 6795 (time 1768142412.01 total 135.86 last 51.83) - open/close 7326 (time 1768142422.44 total 146.29 last 50.90) - open/close 7903 (time 1768142433.19 total 157.04 last 53.65) - open/close 8482 (time 1768142443.95 total 167.80 last 53.84) - open/close 9028 (time 1768142454.44 total 178.29 last 52.03) - open/close 9588 (time 1768142465.19 total 189.04 last 52.08) - open/close 10000 (time 1768142474.10 total 197.95 last 46.27) - open/close 10586 (time 1768142484.71 total 208.56 last 55.21) - open/close 11136 (time 1768142495.33 total 219.19 last 51.77) - open/close 11624 (time 1768142505.34 total 229.19 last 48.79) - open/close 12120 (time 1768142516.32 total 240.18 last 45.14) - open/close 12602 (time 1768142526.33 total 250.18 last 48.19) - open/close 13003 (time 1768142536.33 total 260.18 last 40.09) - open/close 13383 (time 1768142546.41 total 270.26 last 37.71) - open/close 13858 (time 1768142556.65 total 280.50 last 46.39) - open/close 14213 (time 1768142567.40 total 291.26 last 33.00) - open/close 14753 (time 1768142578.15 total 302.00 last 50.25) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14898) error: No space left on device total: 14898 open/close in 303.69 seconds: 49.06 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8925: kill: (111804) - No such process - unlinked 0 (time 1768142585 ; total 0 ; last 0) - unlinked 10000 (time 1768142607 ; total 22 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14898) error: No such file or directory total: 14898 unlinks in 32 seconds: 465.562500 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg438-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg438-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg438-server stop mds service on oleg438-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg438-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg438-server' oleg438-server: oleg438-server.virtnet: executing unload_modules_local oleg438-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg438-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-client: ssh exited with exit code 2 pdsh@oleg438-client: oleg438-server: ssh exited with exit code 2