== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:10:25 (1758741025) start mds service on oleg158-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing load_modules_local oleg158-server: Loading modules from /home/green/git/lustre-release/lustre oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg158-server: detected 4 online CPUs by sysfs oleg158-server: MODOPTS_LIBCFS= oleg158-server: Force libcfs to create 2 CPU partitions oleg158-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg158-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg158-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg158-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg158-server: oleg158-server.virtnet: executing set_default_debug -1 all pdsh@oleg158-client: oleg158-server: ssh exited with exit code 1 Started lustre-OST0000 oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg158-client.virtnet: -o user_xattr,flock 192.168.201.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 305 (time 1758741090.77 total 10.00 last 30.49) - open/close 487 (time 1758741100.81 total 20.05 last 18.13) - open/close 736 (time 1758741110.81 total 30.05 last 24.90) - open/close 950 (time 1758741121.24 total 40.47 last 20.53) - open/close 1114 (time 1758741131.25 total 50.48 last 16.38) - open/close 1408 (time 1758741142.73 total 61.97 last 25.60) - open/close 1690 (time 1758741152.78 total 72.02 last 28.06) - open/close 1968 (time 1758741162.79 total 82.02 last 27.78) - open/close 2276 (time 1758741172.80 total 92.03 last 30.78) - open/close 2665 (time 1758741182.80 total 102.04 last 38.87) - open/close 2965 (time 1758741192.90 total 112.14 last 29.70) - open/close 3136 (time 1758741202.96 total 122.19 last 17.01) - open/close 3337 (time 1758741214.16 total 133.40 last 17.94) - open/close 3569 (time 1758741225.88 total 145.11 last 19.80) - open/close 3829 (time 1758741235.89 total 155.12 last 25.97) - open/close 4124 (time 1758741246.60 total 165.84 last 27.54) - open/close 4450 (time 1758741256.65 total 175.89 last 32.44) - open/close 4715 (time 1758741267.14 total 186.38 last 25.26) - open/close 4986 (time 1758741277.45 total 196.69 last 26.29) - open/close 5299 (time 1758741287.46 total 206.69 last 31.29) - open/close 5580 (time 1758741297.46 total 216.70 last 28.08) - open/close 5762 (time 1758741307.72 total 226.96 last 17.74) - open/close 6054 (time 1758741318.47 total 237.71 last 27.17) - open/close 6295 (time 1758741328.47 total 247.71 last 24.09) - open/close 6541 (time 1758741338.48 total 257.71 last 24.60) - open/close 6781 (time 1758741349.38 total 268.62 last 22.00) - open/close 7041 (time 1758741359.56 total 278.80 last 25.54) - open/close 7308 (time 1758741369.74 total 288.98 last 26.23) - open/close 7578 (time 1758741380.11 total 299.35 last 26.04) - open/close 7865 (time 1758741390.73 total 309.96 last 27.03) - open/close 8136 (time 1758741400.91 total 320.14 last 26.62) - open/close 8398 (time 1758741411.85 total 331.08 last 23.95) - open/close 8627 (time 1758741421.85 total 341.09 last 22.89) - open/close 8884 (time 1758741432.14 total 351.38 last 24.97) - open/close 9195 (time 1758741442.14 total 361.38 last 31.09) - open/close 9491 (time 1758741452.94 total 372.17 last 27.42) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9649) error: No space left on device total: 9649 open/close in 375.79 seconds: 25.68 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8861: kill: (111765) - No such process - unlinked 0 (time 1758741458 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9649) error: No such file or directory total: 9649 unlinks in 31 seconds: 311.258057 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg158-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg158-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg158-server stop mds service on oleg158-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg158-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg158-server' oleg158-server: oleg158-server.virtnet: executing unload_modules_local oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg158-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-client: ssh exited with exit code 2 pdsh@oleg158-client: oleg158-server: ssh exited with exit code 2