== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:30:34 (1777422634) start mds service on oleg102-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing load_modules_local oleg102-server: Loading modules from /home/green/git/lustre-release/lustre oleg102-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg102-server: detected 4 online CPUs by sysfs oleg102-server: MODOPTS_LIBCFS= oleg102-server: Force libcfs to create 2 CPU partitions oleg102-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg102-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg102-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg102-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg102-server: oleg102-server.virtnet: executing set_default_debug -1 all pdsh@oleg102-client: oleg102-server: ssh exited with exit code 1 Started lustre-OST0000 oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg102-client.virtnet: -o user_xattr,flock 192.168.201.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 470 (time 1777422676.33 total 10.00 last 47.00) - open/close 1024 (time 1777422686.62 total 20.29 last 53.83) - open/close 1401 (time 1777422697.37 total 31.04 last 35.07) - open/close 1610 (time 1777422708.83 total 42.50 last 18.24) - open/close 2191 (time 1777422719.52 total 53.19 last 54.35) - open/close 2643 (time 1777422730.40 total 64.07 last 41.55) - open/close 3035 (time 1777422741.27 total 74.95 last 36.03) - open/close 3410 (time 1777422751.78 total 85.45 last 35.71) - open/close 3622 (time 1777422762.21 total 95.88 last 20.33) - open/close 4293 (time 1777422772.70 total 106.37 last 63.95) - open/close 4678 (time 1777422782.82 total 116.49 last 38.04) - open/close 4865 (time 1777422792.86 total 126.53 last 18.62) - open/close 5127 (time 1777422803.29 total 136.96 last 25.12) - open/close 5757 (time 1777422813.40 total 147.08 last 62.30) - open/close 6431 (time 1777422823.70 total 157.38 last 65.42) - open/close 7147 (time 1777422834.33 total 168.00 last 67.40) - open/close 7841 (time 1777422844.33 total 178.00 last 69.39) - open/close 8548 (time 1777422854.36 total 188.03 last 70.47) - open/close 8999 (time 1777422864.73 total 198.40 last 43.51) - open/close 9586 (time 1777422875.42 total 209.09 last 54.91) - open/close 10000 (time 1777422881.62 total 215.29 last 66.77) - open/close 10575 (time 1777422891.74 total 225.41 last 56.83) - open/close 11274 (time 1777422901.74 total 235.41 last 69.88) - open/close 11816 (time 1777422912.15 total 245.83 last 52.05) - open/close 12428 (time 1777422922.33 total 256.00 last 60.13) - open/close 13087 (time 1777422932.33 total 266.01 last 65.89) - open/close 13747 (time 1777422942.33 total 276.01 last 65.99) - open/close 14346 (time 1777422953.12 total 286.79 last 55.56) - open/close 14891 (time 1777422963.67 total 297.34 last 51.63) - open/close 15584 (time 1777422973.67 total 307.35 last 69.29) - open/close 16213 (time 1777422984.41 total 318.08 last 58.59) - open/close 18793 (time 1777422994.41 total 328.08 last 257.98) - open/close 20000 (time 1777422997.26 total 330.94 last 422.74) - open/close 24488 (time 1777423007.26 total 340.94 last 448.76) - open/close 28363 (time 1777423017.27 total 350.94 last 387.43) - open/close 30000 (time 1777423021.04 total 354.71 last 433.94) - open/close 34327 (time 1777423031.04 total 364.71 last 432.69) - open/close 38650 (time 1777423041.04 total 374.71 last 432.26) - open/close 40000 (time 1777423043.88 total 377.55 last 475.12) - open/close 44268 (time 1777423053.89 total 387.56 last 426.64) - open/close 48375 (time 1777423063.89 total 397.56 last 410.68) total: 50000 open/close in 401.39 seconds: 124.57 ops/second - unlinked 0 (time 1777423069 ; total 0 ; last 0) - unlinked 10000 (time 1777423083 ; total 14 ; last 14) - unlinked 20000 (time 1777423097 ; total 28 ; last 14) - unlinked 30000 (time 1777423110 ; total 41 ; last 13) - unlinked 40000 (time 1777423124 ; total 55 ; last 14) total: 50000 unlinks in 70 seconds: 714.285706 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg102-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg102-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg102-server stop mds service on oleg102-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg102-server unloading modules via unload_modules_local on: 'oleg102-server' oleg102-server: oleg102-server.virtnet: executing unload_modules_local oleg102-server: modules unloaded. pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-client: ssh exited with exit code 2 pdsh@oleg102-client: oleg102-server: ssh exited with exit code 2