== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:14:54 (1763352894) start mds service on oleg203-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg203-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 296 (time 1763352953.25 total 12.12 last 24.43) - open/close 426 (time 1763352963.26 total 22.12 last 12.99) - open/close 685 (time 1763352974.11 total 32.98 last 23.86) - open/close 873 (time 1763352984.62 total 43.48 last 17.90) - open/close 1105 (time 1763352995.49 total 54.36 last 21.33) - open/close 1284 (time 1763353005.99 total 64.85 last 17.06) - open/close 1537 (time 1763353016.80 total 75.67 last 23.39) - open/close 1718 (time 1763353027.17 total 86.04 last 17.46) - open/close 1871 (time 1763353037.28 total 96.15 last 15.13) - open/close 2038 (time 1763353047.52 total 106.39 last 16.31) - open/close 2192 (time 1763353057.70 total 116.56 last 15.13) - open/close 2403 (time 1763353068.20 total 127.07 last 20.09) - open/close 2640 (time 1763353078.88 total 137.75 last 22.19) - open/close 3035 (time 1763353088.93 total 147.80 last 39.31) - open/close 3247 (time 1763353099.11 total 157.97 last 20.83) - open/close 3431 (time 1763353109.36 total 168.22 last 17.95) - open/close 3608 (time 1763353119.65 total 178.52 last 17.19) - open/close 3847 (time 1763353132.32 total 191.19 last 18.86) - open/close 4131 (time 1763353143.20 total 202.07 last 26.10) - open/close 4303 (time 1763353153.51 total 212.37 last 16.69) - open/close 4453 (time 1763353163.62 total 222.48 last 14.84) - open/close 4701 (time 1763353174.18 total 233.05 last 23.48) - open/close 4897 (time 1763353184.62 total 243.48 last 18.78) - open/close 5185 (time 1763353195.43 total 254.29 last 26.64) - open/close 5353 (time 1763353205.73 total 264.60 last 16.31) - open/close 5476 (time 1763353215.73 total 274.60 last 12.29) - open/close 5608 (time 1763353226.73 total 285.59 last 12.01) - open/close 5739 (time 1763353237.47 total 296.34 last 12.19) - open/close 6117 (time 1763353247.52 total 306.39 last 37.61) - open/close 6364 (time 1763353258.47 total 317.33 last 22.57) - open/close 6592 (time 1763353268.96 total 327.82 last 21.73) - open/close 6810 (time 1763353279.59 total 338.46 last 20.50) - open/close 7033 (time 1763353290.35 total 349.21 last 20.74) - open/close 7225 (time 1763353300.90 total 359.76 last 18.20) - open/close 7376 (time 1763353311.14 total 370.01 last 14.74) - open/close 7524 (time 1763353321.19 total 380.05 last 14.74) - open/close 8795 (time 1763353331.19 total 390.06 last 127.03) - open/close 10000 (time 1763353337.08 total 395.94 last 204.64) - open/close 12788 (time 1763353347.08 total 405.95 last 278.79) - open/close 14791 (time 1763353357.08 total 415.95 last 200.29) - open/close 17537 (time 1763353367.08 total 425.95 last 274.57) - open/close 20000 (time 1763353376.45 total 435.31 last 262.93) - open/close 22487 (time 1763353386.46 total 445.32 last 248.52) - open/close 24991 (time 1763353396.46 total 455.32 last 250.39) - open/close 26956 (time 1763353406.46 total 465.33 last 196.39) - open/close 28998 (time 1763353416.46 total 475.33 last 204.15) - open/close 30000 (time 1763353422.43 total 481.29 last 168.06) - open/close 31538 (time 1763353432.43 total 491.29 last 153.78) - open/close 33966 (time 1763353442.43 total 501.30 last 242.72) - open/close 35974 (time 1763353452.44 total 511.30 last 200.69) - open/close 38371 (time 1763353462.44 total 521.31 last 239.57) - open/close 40000 (time 1763353467.77 total 526.64 last 305.65) - open/close 42358 (time 1763353477.78 total 536.64 last 235.71) - open/close 44356 (time 1763353487.78 total 546.65 last 199.71) - open/close 47067 (time 1763353497.78 total 556.65 last 271.07) - open/close 49190 (time 1763353507.78 total 566.65 last 212.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 568.29 seconds: 87.34 ops/second - unlinked 0 (time 1763353510 ; total 0 ; last 0) - unlinked 10000 (time 1763353529 ; total 19 ; last 19) - unlinked 20000 (time 1763353550 ; total 40 ; last 21) - unlinked 30000 (time 1763353567 ; total 57 ; last 17) - unlinked 40000 (time 1763353586 ; total 76 ; last 19) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 95 seconds: 522.442078 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg203-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2