== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:33:52 (1786822432) Stopping clients: oleg657-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg657-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg657-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg657-server oleg657-server: oleg657-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg657-server' oleg657-server: oleg657-server.virtnet: executing load_modules_local oleg657-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg657-server: Loading modules from /home/green/git/lustre-release/lustre oleg657-server: detected 4 online CPUs by sysfs oleg657-server: MODOPTS_LIBCFS= oleg657-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg657-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg657-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg657-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg657-server: oleg657-server.virtnet: executing set_default_debug -1 all pdsh@oleg657-client: oleg657-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg657-client.virtnet: -o user_xattr,flock 192.168.206.157@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 951 (time 1786822596.98 total 10.00 last 95.08) - open/close 1914 (time 1786822606.98 total 20.01 last 96.26) - open/close 2869 (time 1786822617.00 total 30.02 last 95.36) - open/close 3674 (time 1786822627.01 total 40.03 last 80.45) total: 4500 open/close in 48.89 seconds: 92.04 ops/second - /unlink 1631 (time 1786822649.89 total 10.00 last 163.09) - /unlink 3030 (time 1786822659.89 total 20.00 last 139.85) total: 4500 /unlink in 29.11 seconds: 154.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 976 (time 1786822683.45 total 10.00 last 97.55) - open/close 1934 (time 1786822693.45 total 20.01 last 95.76) - open/close 2877 (time 1786822703.46 total 30.02 last 94.23) - open/close 3798 (time 1786822713.47 total 40.02 last 92.04) total: 4500 open/close in 47.61 seconds: 94.52 ops/second - /unlink 1825 (time 1786822735.43 total 10.00 last 182.47) - /unlink 3559 (time 1786822745.43 total 20.00 last 173.37) total: 4500 /unlink in 25.41 seconds: 177.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 870 (time 1786822764.70 total 10.02 last 86.82) - open/close 1683 (time 1786822774.71 total 20.03 last 81.20) - open/close 2509 (time 1786822784.72 total 30.04 last 82.57) - open/close 3374 (time 1786822794.72 total 40.04 last 86.48) - open/close 4269 (time 1786822804.73 total 50.05 last 89.44) total: 4500 open/close in 52.83 seconds: 85.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1762 (time 1786822821.48 total 10.00 last 176.18) - /unlink 3371 (time 1786822831.49 total 20.01 last 160.71) total: 4500 /unlink in 27.29 seconds: 164.92 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 838 (time 1786822853.55 total 10.03 last 83.58) - open/close 1743 (time 1786822863.55 total 20.03 last 90.48) - open/close 2554 (time 1786822873.57 total 30.04 last 81.01) - open/close 3413 (time 1786822883.57 total 40.05 last 85.84) - open/close 4278 (time 1786822893.58 total 50.05 last 86.43) total: 4500 open/close in 52.51 seconds: 85.70 ops/second - /unlink 1731 (time 1786822909.53 total 10.00 last 173.06) - /unlink 3426 (time 1786822919.53 total 20.00 last 169.49) total: 4500 /unlink in 26.57 seconds: 169.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 871 (time 1786822940.01 total 10.02 last 86.97) - open/close 1720 (time 1786822950.01 total 20.02 last 84.86) - open/close 2540 (time 1786822960.02 total 30.02 last 81.98) - open/close 3407 (time 1786822970.03 total 40.03 last 86.59) - open/close 4378 (time 1786822980.03 total 50.04 last 97.08) total: 4500 open/close in 51.40 seconds: 87.54 ops/second - /unlink 1728 (time 1786822995.22 total 10.00 last 172.75) - /unlink 3497 (time 1786823005.22 total 20.00 last 176.87) total: 4500 /unlink in 26.51 seconds: 169.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 963 (time 1786823025.43 total 10.01 last 96.25) - open/close 1908 (time 1786823035.45 total 20.02 last 94.39) - open/close 2830 (time 1786823045.45 total 30.02 last 92.20) - open/close 3683 (time 1786823055.46 total 40.03 last 85.21) total: 4500 open/close in 49.20 seconds: 91.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1622 (time 1786823079.65 total 10.00 last 162.16) - /unlink 3334 (time 1786823089.66 total 20.00 last 171.18) total: 4500 /unlink in 26.85 seconds: 167.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 931 (time 1786823110.52 total 10.00 last 93.09) - open/close 1873 (time 1786823120.54 total 20.01 last 94.08) - open/close 2823 (time 1786823130.55 total 30.02 last 94.91) - open/close 3739 (time 1786823140.55 total 40.03 last 91.57) total: 4500 open/close in 48.13 seconds: 93.49 ops/second - /unlink 1620 (time 1786823162.21 total 10.00 last 161.92) - /unlink 3333 (time 1786823172.22 total 20.01 last 171.27) total: 4500 /unlink in 27.03 seconds: 166.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 773 (time 1786823194.10 total 10.01 last 77.25) - open/close 1634 (time 1786823204.10 total 20.01 last 86.09) - open/close 2572 (time 1786823214.10 total 30.01 last 93.79) - open/close 3537 (time 1786823224.10 total 40.01 last 96.47) - open/close 4466 (time 1786823234.11 total 50.01 last 92.87) total: 4500 open/close in 50.38 seconds: 89.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1594 (time 1786823248.43 total 10.00 last 159.33) - /unlink 3327 (time 1786823258.44 total 20.01 last 173.22) total: 4500 /unlink in 27.78 seconds: 162.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 944 (time 1786823280.67 total 10.01 last 94.30) - open/close 1772 (time 1786823290.68 total 20.02 last 82.73) - open/close 2570 (time 1786823300.68 total 30.02 last 79.77) - open/close 3329 (time 1786823310.69 total 40.03 last 75.83) - open/close 4252 (time 1786823320.70 total 50.04 last 92.26) total: 4500 open/close in 52.48 seconds: 85.75 ops/second - /unlink 1711 (time 1786823337.26 total 10.01 last 171.01) - /unlink 3396 (time 1786823347.26 total 20.01 last 168.40) total: 4500 /unlink in 26.83 seconds: 167.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 884 (time 1786823368.04 total 10.01 last 88.30) - open/close 1780 (time 1786823378.05 total 20.02 last 89.50) - open/close 2681 (time 1786823388.05 total 30.03 last 90.07) - open/close 3630 (time 1786823398.06 total 40.03 last 94.85) total: 4500 open/close in 49.53 seconds: 90.86 ops/second - /unlink 1739 (time 1786823421.21 total 10.01 last 173.80) - /unlink 3334 (time 1786823431.22 total 20.01 last 159.45) total: 4500 /unlink in 28.13 seconds: 159.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 988 (time 1786823453.88 total 10.00 last 98.78) - open/close 1931 (time 1786823463.89 total 20.01 last 94.21) - open/close 2824 (time 1786823473.89 total 30.02 last 89.23) - open/close 3777 (time 1786823483.90 total 40.02 last 95.27) total: 4500 open/close in 47.13 seconds: 95.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1772 (time 1786823504.73 total 10.00 last 177.12) - /unlink 3399 (time 1786823514.74 total 20.01 last 162.56) total: 4500 /unlink in 26.03 seconds: 172.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 930 (time 1786823534.73 total 10.00 last 93.00) - open/close 1831 (time 1786823544.73 total 20.01 last 90.05) - open/close 2791 (time 1786823554.74 total 30.01 last 95.95) - open/close 3693 (time 1786823564.74 total 40.01 last 90.20) total: 4500 open/close in 49.39 seconds: 91.10 ops/second - /unlink 1615 (time 1786823588.37 total 10.00 last 161.44) - /unlink 2844 (time 1786823598.38 total 20.01 last 122.85) - /unlink 4301 (time 1786823608.38 total 30.01 last 145.63) total: 4500 /unlink in 31.39 seconds: 143.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 974 (time 1786823624.19 total 10.00 last 97.36) - open/close 1778 (time 1786823634.20 total 20.01 last 80.34) - open/close 2714 (time 1786823644.20 total 30.01 last 93.58) - open/close 3697 (time 1786823654.22 total 40.03 last 98.10) total: 4500 open/close in 48.34 seconds: 93.08 ops/second - /unlink 1716 (time 1786823676.42 total 10.00 last 171.56) - /unlink 3262 (time 1786823686.42 total 20.01 last 154.48) total: 4500 /unlink in 28.33 seconds: 158.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 829 (time 1786823708.66 total 10.00 last 82.90) - open/close 1693 (time 1786823718.66 total 20.00 last 86.40) - open/close 2598 (time 1786823728.67 total 30.01 last 90.44) - open/close 3501 (time 1786823738.67 total 40.01 last 90.27) - open/close 4383 (time 1786823748.69 total 50.03 last 88.04) total: 4500 open/close in 51.61 seconds: 87.19 ops/second - /unlink 1760 (time 1786823764.27 total 10.00 last 175.99) - /unlink 3396 (time 1786823774.27 total 20.00 last 163.54) total: 4500 /unlink in 26.66 seconds: 168.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 984 (time 1786823794.68 total 10.01 last 98.32) - open/close 1938 (time 1786823804.69 total 20.02 last 95.30) - open/close 2870 (time 1786823814.69 total 30.02 last 93.18) - open/close 3692 (time 1786823824.71 total 40.04 last 82.06) total: 4500 open/close in 49.51 seconds: 90.89 ops/second - /unlink 1603 (time 1786823847.80 total 10.01 last 160.18) - /unlink 3275 (time 1786823857.81 total 20.01 last 167.16) total: 4500 /unlink in 27.47 seconds: 163.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 886 (time 1786823879.66 total 10.01 last 88.53) - open/close 1730 (time 1786823889.66 total 20.01 last 84.35) - open/close 2663 (time 1786823899.66 total 30.02 last 93.29) - open/close 3524 (time 1786823909.68 total 40.03 last 85.96) - open/close 4462 (time 1786823919.69 total 50.04 last 93.75) total: 4500 open/close in 50.36 seconds: 89.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1722 (time 1786823933.72 total 10.01 last 172.10) - /unlink 3361 (time 1786823943.72 total 20.01 last 163.83) total: 4500 /unlink in 27.32 seconds: 164.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 971 (time 1786823965.77 total 10.00 last 97.07) - open/close 1908 (time 1786823975.81 total 20.04 last 93.39) - open/close 2812 (time 1786823985.82 total 30.05 last 90.30) - open/close 3704 (time 1786823995.82 total 40.05 last 89.15) total: 4500 open/close in 49.27 seconds: 91.32 ops/second - /unlink 1597 (time 1786824019.30 total 10.00 last 159.70) - /unlink 3194 (time 1786824029.30 total 20.00 last 159.66) total: 4500 /unlink in 28.01 seconds: 160.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 954 (time 1786824051.79 total 10.00 last 95.38) - open/close 1898 (time 1786824061.82 total 20.03 last 94.15) - open/close 2844 (time 1786824071.82 total 30.03 last 94.58) - open/close 3728 (time 1786824081.82 total 40.04 last 88.34) total: 4500 open/close in 48.46 seconds: 92.85 ops/second - /unlink 1570 (time 1786824104.45 total 10.00 last 156.99) - /unlink 3159 (time 1786824114.45 total 20.00 last 158.88) total: 4500 /unlink in 27.99 seconds: 160.77 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 94743 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg657-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg657-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg657-server stop mds service on oleg657-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg657-server stop mds service on oleg657-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg657-server unloading modules via unload_modules_local on: 'oleg657-server' oleg657-server: oleg657-server.virtnet: executing unload_modules_local oleg657-server: modules unloaded.