== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:23:23 (1762169003) start mds service on oleg247-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg247-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg247-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg247-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 151 (time 1762169058.25 total 10.03 last 15.05) - open/close 287 (time 1762169068.25 total 20.03 last 13.60) - open/close 502 (time 1762169079.37 total 31.15 last 19.34) - open/close 742 (time 1762169090.20 total 41.98 last 22.16) - open/close 928 (time 1762169100.68 total 52.46 last 17.74) - open/close 1179 (time 1762169111.38 total 63.16 last 23.47) - open/close 1414 (time 1762169122.21 total 74.00 last 21.68) - open/close 1623 (time 1762169132.22 total 84.01 last 20.88) - open/close 1844 (time 1762169143.19 total 94.98 last 20.15) - open/close 2113 (time 1762169154.02 total 105.80 last 24.84) - open/close 2382 (time 1762169164.49 total 116.28 last 25.69) - open/close 2587 (time 1762169175.12 total 126.91 last 19.28) - open/close 2749 (time 1762169188.19 total 139.97 last 12.40) - open/close 2858 (time 1762169198.67 total 150.45 last 10.40) - open/close 3160 (time 1762169209.24 total 161.02 last 28.58) - open/close 3255 (time 1762169219.74 total 171.52 last 9.05) - open/close 3379 (time 1762169230.35 total 182.13 last 11.68) - open/close 3508 (time 1762169241.42 total 193.21 last 11.65) - open/close 3641 (time 1762169252.30 total 204.08 last 12.23) - open/close 3819 (time 1762169262.61 total 214.39 last 17.27) - open/close 4117 (time 1762169272.61 total 224.39 last 29.80) - open/close 4268 (time 1762169283.16 total 234.94 last 14.31) - open/close 4408 (time 1762169293.20 total 244.99 last 13.93) - open/close 4552 (time 1762169304.01 total 255.79 last 13.33) - open/close 4828 (time 1762169314.77 total 266.55 last 25.65) - open/close 5048 (time 1762169325.14 total 276.92 last 21.21) - open/close 5177 (time 1762169335.84 total 287.62 last 12.06) - open/close 5503 (time 1762169345.84 total 297.62 last 32.59) - open/close 5695 (time 1762169355.85 total 307.63 last 19.19) - open/close 6006 (time 1762169366.80 total 318.58 last 28.40) - open/close 6257 (time 1762169377.29 total 329.07 last 23.92) - open/close 6411 (time 1762169389.66 total 341.44 last 12.45) - open/close 6542 (time 1762169400.47 total 352.25 last 12.12) - open/close 6775 (time 1762169411.03 total 362.81 last 22.06) - open/close 6977 (time 1762169421.47 total 373.25 last 19.35) - open/close 7194 (time 1762169431.69 total 383.47 last 21.23) - open/close 7491 (time 1762169442.38 total 394.16 last 27.78) - open/close 7835 (time 1762169453.33 total 405.11 last 31.41) - open/close 8140 (time 1762169464.01 total 415.79 last 28.56) - open/close 8368 (time 1762169474.06 total 425.84 last 22.70) - open/close 8641 (time 1762169484.37 total 436.15 last 26.48) - open/close 8916 (time 1762169494.60 total 446.39 last 26.86) - open/close 9182 (time 1762169504.97 total 456.75 last 25.65) - open/close 9417 (time 1762169515.72 total 467.50 last 21.87) - open/close 9683 (time 1762169525.72 total 477.50 last 26.59) - open/close 9908 (time 1762169536.46 total 488.24 last 20.96) - open/close 10000 (time 1762169539.07 total 490.85 last 35.20) - open/close 10558 (time 1762169549.07 total 500.85 last 55.78) - open/close 11966 (time 1762169559.08 total 510.86 last 140.73) - open/close 13416 (time 1762169569.08 total 520.86 last 144.95) - open/close 14564 (time 1762169579.08 total 530.87 last 114.77) - open/close 15991 (time 1762169589.09 total 540.87 last 142.65) - open/close 17354 (time 1762169599.09 total 550.88 last 136.22) - open/close 18566 (time 1762169609.10 total 560.88 last 121.09) - open/close 19858 (time 1762169619.11 total 570.89 last 129.17) - open/close 21288 (time 1762169629.11 total 580.89 last 142.90) - open/close 22463 (time 1762169639.11 total 590.89 last 117.49) - open/close 23709 (time 1762169649.12 total 600.91 last 124.47) - open/close 24897 (time 1762169659.12 total 610.91 last 118.78) - open/close 25957 (time 1762169669.13 total 620.91 last 105.94) - open/close 27127 (time 1762169679.13 total 630.92 last 116.95) - open/close 28307 (time 1762169689.14 total 640.92 last 117.95) - open/close 29527 (time 1762169699.14 total 650.93 last 121.92) - open/close 30000 (time 1762169702.12 total 653.90 last 158.94) - open/close 31635 (time 1762169712.13 total 663.91 last 163.38) - open/close 32902 (time 1762169722.13 total 673.91 last 126.70) - open/close 34059 (time 1762169732.13 total 683.92 last 115.63) - open/close 35239 (time 1762169742.14 total 693.92 last 117.94) - open/close 36550 (time 1762169752.14 total 703.92 last 131.08) - open/close 38011 (time 1762169762.15 total 713.93 last 146.02) - open/close 39442 (time 1762169772.15 total 723.94 last 143.00) - open/close 40000 (time 1762169775.86 total 727.64 last 150.52) - open/close 41229 (time 1762169785.86 total 737.64 last 122.87) - open/close 42815 (time 1762169795.87 total 747.65 last 158.51) - open/close 44152 (time 1762169805.87 total 757.65 last 133.64) - open/close 45457 (time 1762169815.87 total 767.66 last 130.48) - open/close 46862 (time 1762169825.88 total 777.66 last 140.48) - open/close 48223 (time 1762169835.88 total 787.66 last 136.08) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 797.41 seconds: 62.24 ops/second - unlinked 0 (time 1762169847 ; total 0 ; last 0) - unlinked 10000 (time 1762169890 ; total 43 ; last 43) - unlinked 20000 (time 1762169930 ; total 83 ; last 40) - unlinked 30000 (time 1762169958 ; total 111 ; last 28) - unlinked 40000 (time 1762169992 ; total 145 ; last 34) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 171 seconds: 290.245605 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local oleg247-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg247-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2