== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:54:48 (1772438088) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg440-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1772438144.65 total 10.40 last 22.89) - open/close 570 (time 1772438155.34 total 21.09 last 31.06) - open/close 1007 (time 1772438165.35 total 31.09 last 43.69) - open/close 1660 (time 1772438175.76 total 41.50 last 62.72) - open/close 2034 (time 1772438186.51 total 52.25 last 34.78) - open/close 2560 (time 1772438196.69 total 62.43 last 51.69) - open/close 2715 (time 1772438206.87 total 72.61 last 15.22) - open/close 2889 (time 1772438217.30 total 83.04 last 16.68) - open/close 3092 (time 1772438227.86 total 93.60 last 19.22) - open/close 3336 (time 1772438238.29 total 104.03 last 23.40) - open/close 3537 (time 1772438248.59 total 114.34 last 19.50) - open/close 3820 (time 1772438258.89 total 124.64 last 27.47) - open/close 4321 (time 1772438268.89 total 134.64 last 50.10) - open/close 4657 (time 1772438279.57 total 145.31 last 31.48) - open/close 4977 (time 1772438290.19 total 155.93 last 30.12) - open/close 5312 (time 1772438300.81 total 166.56 last 31.54) - open/close 5512 (time 1772438311.31 total 177.06 last 19.05) - open/close 5876 (time 1772438322.20 total 187.94 last 33.45) - open/close 6410 (time 1772438332.50 total 198.24 last 51.84) - open/close 6735 (time 1772438343.25 total 208.99 last 30.23) - open/close 7361 (time 1772438353.74 total 219.48 last 59.66) - open/close 7958 (time 1772438364.11 total 229.85 last 57.58) - open/close 8283 (time 1772438376.02 total 241.76 last 27.29) - open/close 8625 (time 1772438386.77 total 252.51 last 31.81) - open/close 8804 (time 1772438396.88 total 262.62 last 17.71) - open/close 9262 (time 1772438407.89 total 273.63 last 41.60) - open/close 9573 (time 1772438418.45 total 284.19 last 29.45) - open/close 9860 (time 1772438428.95 total 294.69 last 27.34) - open/close 9992 (time 1772438439.96 total 305.70 last 11.99) - open/close 10431 (time 1772438449.96 total 315.70 last 43.90) - open/close 10665 (time 1772438460.37 total 326.11 last 22.49) - open/close 11107 (time 1772438470.37 total 336.11 last 44.19) - open/close 11400 (time 1772438481.04 total 346.78 last 27.46) - open/close 13234 (time 1772438491.05 total 356.79 last 183.26) - open/close 16670 (time 1772438501.05 total 366.80 last 343.46) - open/close 19331 (time 1772438511.05 total 376.80 last 266.05) - open/close 23019 (time 1772438521.06 total 386.80 last 368.78) - open/close 25966 (time 1772438531.06 total 396.80 last 294.68) - open/close 28745 (time 1772438541.06 total 406.80 last 277.81) - open/close 30000 (time 1772438546.28 total 412.03 last 240.18) - open/close 33308 (time 1772438556.29 total 422.03 last 330.78) - open/close 36801 (time 1772438566.29 total 432.03 last 349.29) - open/close 40000 (time 1772438575.34 total 441.08 last 353.48) - open/close 42720 (time 1772438585.34 total 451.08 last 271.90) - open/close 46382 (time 1772438595.34 total 461.08 last 366.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 470.02 seconds: 105.60 ops/second - unlinked 0 (time 1772438605 ; total 0 ; last 0) - unlinked 10000 (time 1772438619 ; total 14 ; last 14) - unlinked 20000 (time 1772438635 ; total 30 ; last 16) - unlinked 30000 (time 1772438650 ; total 45 ; last 15) - unlinked 40000 (time 1772438664 ; total 59 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 72 seconds: 689.333313 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2