== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:15:02 (1772475302) start mds service on oleg453-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg453-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1772475343.17 total 10.63 last 19.28) - open/close 536 (time 1772475353.18 total 20.64 last 33.08) - open/close 756 (time 1772475363.78 total 31.24 last 20.75) - open/close 990 (time 1772475374.66 total 42.12 last 21.50) - open/close 1240 (time 1772475385.67 total 53.13 last 22.71) - open/close 1477 (time 1772475395.68 total 63.14 last 23.68) - open/close 1803 (time 1772475406.35 total 73.81 last 30.55) - open/close 2072 (time 1772475416.53 total 83.99 last 26.43) - open/close 2375 (time 1772475427.08 total 94.54 last 28.71) - open/close 2540 (time 1772475437.45 total 104.91 last 15.92) - open/close 2825 (time 1772475448.26 total 115.72 last 26.35) - open/close 3159 (time 1772475459.08 total 126.54 last 30.88) - open/close 3356 (time 1772475469.58 total 137.04 last 18.76) - open/close 3624 (time 1772475479.94 total 147.41 last 25.85) - open/close 3903 (time 1772475489.99 total 157.45 last 27.78) - open/close 4166 (time 1772475500.10 total 167.56 last 26.01) - open/close 4294 (time 1772475511.11 total 178.57 last 11.62) - open/close 4583 (time 1772475521.54 total 189.01 last 27.70) - open/close 4844 (time 1772475531.59 total 199.05 last 25.98) - open/close 5211 (time 1772475542.47 total 209.93 last 33.73) - open/close 5458 (time 1772475552.47 total 219.94 last 24.69) - open/close 5680 (time 1772475563.41 total 230.87 last 20.30) - open/close 5854 (time 1772475573.96 total 241.42 last 16.50) - open/close 6045 (time 1772475584.58 total 252.04 last 17.98) - open/close 6303 (time 1772475595.14 total 262.60 last 24.43) - open/close 6628 (time 1772475605.58 total 273.04 last 31.14) - open/close 6882 (time 1772475616.13 total 283.60 last 24.06) - open/close 7261 (time 1772475627.02 total 294.48 last 34.83) - open/close 7512 (time 1772475639.56 total 307.02 last 20.01) - open/close 7827 (time 1772475650.24 total 317.71 last 29.48) - open/close 8512 (time 1772475660.74 total 328.20 last 65.25) - open/close 9112 (time 1772475671.56 total 339.02 last 55.48) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9361) error: No space left on device total: 9361 open/close in 341.98 seconds: 27.37 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8955: kill: (74292) - No such process - unlinked 0 (time 1772475676 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-9361) error: No such file or directory total: 9361 unlinks in 19 seconds: 492.684204 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2