== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:12 (1773237552) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg424-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 787 (time 1773237608.99 total 10.15 last 77.50) - open/close 1516 (time 1773237618.99 total 20.16 last 72.88) - open/close 2314 (time 1773237629.02 total 30.19 last 79.57) - open/close 3136 (time 1773237639.13 total 40.30 last 81.27) - open/close 3988 (time 1773237649.38 total 50.54 last 83.17) - open/close 4846 (time 1773237659.61 total 60.78 last 83.83) - open/close 5650 (time 1773237669.72 total 70.89 last 79.52) - open/close 6419 (time 1773237680.73 total 81.90 last 69.85) - open/close 7200 (time 1773237690.73 total 91.90 last 78.09) - open/close 7922 (time 1773237701.53 total 102.70 last 66.86) - open/close 8680 (time 1773237711.53 total 112.70 last 75.80) - open/close 9463 (time 1773237722.52 total 123.69 last 71.25) - open/close 10000 (time 1773237728.83 total 129.99 last 85.21) - open/close 10741 (time 1773237739.74 total 140.91 last 67.88) - open/close 11536 (time 1773237749.79 total 150.96 last 79.12) - open/close 12293 (time 1773237759.79 total 160.96 last 75.69) - open/close 13027 (time 1773237769.79 total 170.96 last 73.39) - open/close 13814 (time 1773237779.81 total 180.98 last 78.57) - open/close 14568 (time 1773237790.68 total 191.85 last 69.33) - open/close 15330 (time 1773237800.68 total 201.85 last 76.20) - open/close 15886 (time 1773237811.16 total 212.33 last 53.06) - open/close 16291 (time 1773237821.28 total 222.45 last 40.03) - open/close 16936 (time 1773237832.03 total 233.19 last 60.01) - open/close 17749 (time 1773237843.04 total 244.20 last 73.85) - open/close 18547 (time 1773237853.04 total 254.20 last 79.79) - open/close 19188 (time 1773237863.84 total 265.01 last 59.34) - open/close 19979 (time 1773237873.84 total 275.01 last 79.09) - open/close 20688 (time 1773237884.70 total 285.87 last 65.29) - open/close 21534 (time 1773237894.88 total 296.05 last 83.12) - open/close 22384 (time 1773237904.88 total 306.05 last 85.00) - open/close 26135 (time 1773237914.88 total 316.05 last 375.04) - open/close 30000 (time 1773237923.27 total 324.44 last 460.50) - open/close 34703 (time 1773237933.28 total 334.44 last 470.26) - open/close 40000 (time 1773237942.62 total 343.79 last 566.56) - open/close 45340 (time 1773237952.64 total 353.81 last 533.22) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 362.07 seconds: 137.08 ops/second - unlinked 0 (time 1773237962 ; total 0 ; last 0) - unlinked 10000 (time 1773237972 ; total 10 ; last 10) - unlinked 20000 (time 1773237983 ; total 21 ; last 11) - unlinked 30000 (time 1773237995 ; total 33 ; last 12) - unlinked 40000 (time 1773238007 ; total 45 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 55 seconds: 902.400024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg424-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2