== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:38:27 (1786747107) start mds service on oleg459-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg459-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 156 (time 1786747200.00 total 10.51 last 14.84) - open/close 365 (time 1786747210.10 total 20.62 last 20.68) - open/close 559 (time 1786747220.66 total 31.17 last 18.38) - open/close 704 (time 1786747231.02 total 41.53 last 13.99) - open/close 928 (time 1786747241.26 total 51.78 last 21.87) - open/close 1044 (time 1786747251.93 total 62.44 last 10.88) - open/close 1213 (time 1786747262.67 total 73.18 last 15.73) - open/close 1279 (time 1786747272.77 total 83.28 last 6.54) - open/close 1471 (time 1786747285.24 total 95.76 last 15.39) - open/close 1674 (time 1786747296.12 total 106.63 last 18.67) - open/close 1901 (time 1786747306.56 total 117.07 last 21.74) - open/close 2034 (time 1786747317.49 total 128.00 last 12.16) - open/close 2207 (time 1786747327.86 total 138.37 last 16.69) - open/close 2331 (time 1786747338.76 total 149.27 last 11.38) - open/close 2522 (time 1786747349.29 total 159.80 last 18.14) - open/close 2674 (time 1786747359.30 total 169.81 last 15.18) - open/close 2847 (time 1786747369.31 total 179.82 last 17.29) - open/close 3094 (time 1786747380.09 total 190.60 last 22.91) - open/close 3323 (time 1786747390.67 total 201.18 last 21.64) - open/close 3512 (time 1786747400.68 total 211.20 last 18.87) - open/close 3762 (time 1786747411.26 total 221.77 last 23.64) - open/close 4035 (time 1786747422.00 total 232.52 last 25.41) - open/close 4200 (time 1786747432.89 total 243.40 last 15.16) - open/close 4272 (time 1786747444.92 total 255.43 last 5.99) - open/close 4496 (time 1786747455.31 total 265.82 last 21.56) - open/close 4825 (time 1786747466.06 total 276.57 last 30.60) - open/close 5045 (time 1786747476.47 total 286.98 last 21.13) - open/close 5291 (time 1786747486.84 total 297.36 last 23.72) - open/close 5562 (time 1786747497.84 total 308.36 last 24.63) - open/close 5779 (time 1786747508.82 total 319.34 last 19.76) - open/close 5998 (time 1786747518.83 total 329.35 last 21.88) - open/close 6178 (time 1786747528.97 total 339.48 last 17.76) - open/close 6296 (time 1786747538.98 total 349.50 last 11.78) - open/close 6322 (time 1786747549.96 total 360.48 last 2.37) - open/close 6462 (time 1786747560.62 total 371.14 last 13.13) - open/close 6658 (time 1786747572.87 total 383.38 last 16.00) - open/close 6798 (time 1786747583.09 total 393.60 last 13.71) - open/close 6864 (time 1786747593.12 total 403.63 last 6.58) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6865) error: No space left on device total: 6865 open/close in 403.68 seconds: 17.01 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76077) - No such process - unlinked 0 (time 1786747607 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6865) error: No such file or directory total: 6865 unlinks in 63 seconds: 108.968254 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: modules unloaded. pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2