== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:22:10 (1787628130) Stopping clients: oleg311-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg311-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg311-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg311-server oleg311-server: oleg311-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg311-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg311-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg311-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 789 (time 1787628340.97 total 10.01 last 78.85) - open/close 1506 (time 1787628350.98 total 20.02 last 71.63) - open/close 2336 (time 1787628360.98 total 30.02 last 82.99) - open/close 3108 (time 1787628370.99 total 40.03 last 77.13) - open/close 3874 (time 1787628381.00 total 50.04 last 76.53) total: 4500 open/close in 57.50 seconds: 78.26 ops/second - /unlink 1393 (time 1787628402.54 total 10.00 last 139.26) - /unlink 2706 (time 1787628412.55 total 20.01 last 131.20) - /unlink 4249 (time 1787628422.55 total 30.01 last 154.26) total: 4500 /unlink in 31.48 seconds: 142.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 923 (time 1787628437.73 total 10.01 last 92.25) - open/close 1784 (time 1787628447.73 total 20.01 last 86.08) - open/close 2620 (time 1787628457.73 total 30.02 last 83.54) - open/close 3422 (time 1787628467.74 total 40.02 last 80.17) - open/close 4276 (time 1787628477.75 total 50.03 last 85.33) total: 4500 open/close in 52.55 seconds: 85.63 ops/second - /unlink 1456 (time 1787628494.11 total 10.01 last 145.52) - /unlink 2963 (time 1787628504.11 total 20.01 last 150.65) - /unlink 4464 (time 1787628514.12 total 30.01 last 150.04) total: 4500 /unlink in 30.28 seconds: 148.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 827 (time 1787628527.83 total 10.02 last 82.52) - open/close 1752 (time 1787628537.84 total 20.03 last 92.45) - open/close 2506 (time 1787628547.84 total 30.03 last 75.39) - open/close 3345 (time 1787628557.85 total 40.04 last 83.83) - open/close 4191 (time 1787628567.85 total 50.04 last 84.58) total: 4500 open/close in 54.07 seconds: 83.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1505 (time 1787628586.04 total 10.00 last 150.45) - /unlink 3044 (time 1787628596.04 total 20.01 last 153.83) total: 4500 /unlink in 28.92 seconds: 155.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 835 (time 1787628619.40 total 10.00 last 83.46) - open/close 1654 (time 1787628629.40 total 20.01 last 81.89) - open/close 2454 (time 1787628639.40 total 30.01 last 79.97) - open/close 3256 (time 1787628649.41 total 40.02 last 80.12) - open/close 4083 (time 1787628659.42 total 50.03 last 82.66) total: 4500 open/close in 54.54 seconds: 82.50 ops/second - /unlink 1420 (time 1787628678.01 total 10.00 last 141.99) - /unlink 2698 (time 1787628688.02 total 20.01 last 127.67) - /unlink 4267 (time 1787628698.02 total 30.01 last 156.86) total: 4500 /unlink in 31.90 seconds: 141.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 814 (time 1787628714.23 total 10.00 last 81.36) - open/close 1611 (time 1787628724.24 total 20.00 last 79.70) - open/close 2362 (time 1787628734.26 total 30.03 last 74.92) - open/close 3207 (time 1787628744.27 total 40.04 last 84.45) - open/close 4037 (time 1787628754.27 total 50.04 last 82.99) total: 4500 open/close in 55.45 seconds: 81.15 ops/second - /unlink 1364 (time 1787628774.07 total 10.00 last 136.39) - /unlink 2710 (time 1787628784.08 total 20.01 last 134.49) - /unlink 3948 (time 1787628794.09 total 30.02 last 123.72) total: 4500 /unlink in 33.94 seconds: 132.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 754 (time 1787628811.58 total 10.00 last 75.38) - open/close 1531 (time 1787628821.59 total 20.01 last 77.61) - open/close 2322 (time 1787628831.60 total 30.02 last 79.03) - open/close 3034 (time 1787628841.61 total 40.03 last 71.14) - open/close 3772 (time 1787628851.61 total 50.04 last 73.76) total: 4500 open/close in 58.66 seconds: 76.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1533 (time 1787628874.12 total 10.02 last 153.03) - /unlink 2992 (time 1787628884.12 total 20.02 last 145.87) - /unlink 4185 (time 1787628894.12 total 30.02 last 119.26) total: 4500 /unlink in 33.05 seconds: 136.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 502 (time 1787628913.94 total 10.02 last 50.09) - open/close 1037 (time 1787628923.96 total 20.04 last 53.39) - open/close 1723 (time 1787628933.97 total 30.05 last 68.55) - open/close 2547 (time 1787628943.97 total 40.05 last 82.38) - open/close 3361 (time 1787628953.97 total 50.06 last 81.35) - open/close 4181 (time 1787628964.01 total 60.09 last 81.72) total: 4500 open/close in 64.60 seconds: 69.66 ops/second - /unlink 1352 (time 1787628982.43 total 10.03 last 134.86) - /unlink 2424 (time 1787628992.43 total 20.03 last 107.20) - /unlink 3746 (time 1787629002.43 total 30.03 last 132.17) total: 4500 /unlink in 35.04 seconds: 128.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 756 (time 1787629022.10 total 10.01 last 75.55) - open/close 1491 (time 1787629032.10 total 20.01 last 73.48) - open/close 2209 (time 1787629042.10 total 30.02 last 71.76) - open/close 2936 (time 1787629052.11 total 40.02 last 72.65) - open/close 3689 (time 1787629062.11 total 50.02 last 75.28) total: 4500 open/close in 59.83 seconds: 75.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1535 (time 1787629085.67 total 10.00 last 153.45) - /unlink 3068 (time 1787629095.67 total 20.01 last 153.27) total: 4500 /unlink in 29.46 seconds: 152.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 797 (time 1787629120.19 total 10.01 last 79.61) - open/close 1638 (time 1787629130.20 total 20.02 last 84.01) - open/close 2402 (time 1787629140.21 total 30.03 last 76.32) - open/close 3149 (time 1787629150.22 total 40.05 last 74.61) - open/close 3992 (time 1787629160.23 total 50.05 last 84.26) total: 4500 open/close in 56.20 seconds: 80.07 ops/second - /unlink 1480 (time 1787629180.53 total 10.00 last 147.99) - /unlink 2899 (time 1787629190.53 total 20.00 last 141.90) - /unlink 4421 (time 1787629200.54 total 30.01 last 152.12) total: 4500 /unlink in 30.79 seconds: 146.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 859 (time 1787629215.60 total 10.00 last 85.88) - open/close 1629 (time 1787629225.62 total 20.02 last 76.88) - open/close 2460 (time 1787629235.63 total 30.03 last 83.00) - open/close 3355 (time 1787629245.64 total 40.04 last 89.42) - open/close 4212 (time 1787629255.64 total 50.04 last 85.67) total: 4500 open/close in 53.79 seconds: 83.66 ops/second - /unlink 1678 (time 1787629273.11 total 10.00 last 167.78) - /unlink 3075 (time 1787629283.11 total 20.01 last 139.64) - /unlink 4467 (time 1787629293.11 total 30.01 last 139.17) total: 4500 /unlink in 30.29 seconds: 148.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 819 (time 1787629307.77 total 10.00 last 81.88) - open/close 1670 (time 1787629317.79 total 20.02 last 84.97) - open/close 2493 (time 1787629327.80 total 30.02 last 82.24) - open/close 3355 (time 1787629337.81 total 40.04 last 86.10) - open/close 4196 (time 1787629347.82 total 50.04 last 84.03) total: 4500 open/close in 53.12 seconds: 84.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1711 (time 1787629364.09 total 10.00 last 171.07) - /unlink 3287 (time 1787629374.10 total 20.00 last 157.55) total: 4500 /unlink in 27.26 seconds: 165.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 745 (time 1787629396.34 total 10.03 last 74.31) - open/close 1428 (time 1787629406.34 total 20.03 last 68.29) - open/close 2326 (time 1787629416.35 total 30.03 last 89.76) - open/close 3178 (time 1787629426.35 total 40.04 last 85.17) - open/close 4079 (time 1787629436.36 total 50.04 last 90.07) total: 4500 open/close in 54.92 seconds: 81.94 ops/second - /unlink 1459 (time 1787629454.94 total 10.00 last 145.88) - /unlink 3005 (time 1787629464.94 total 20.01 last 154.52) total: 4500 /unlink in 29.81 seconds: 150.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 892 (time 1787629488.67 total 10.02 last 89.05) - open/close 1708 (time 1787629498.67 total 20.02 last 81.60) - open/close 2479 (time 1787629508.69 total 30.04 last 76.95) - open/close 3328 (time 1787629518.70 total 40.04 last 84.86) - open/close 4158 (time 1787629528.71 total 50.05 last 82.90) total: 4500 open/close in 53.73 seconds: 83.74 ops/second - /unlink 1666 (time 1787629545.89 total 10.00 last 166.56) - /unlink 3233 (time 1787629555.90 total 20.01 last 156.64) total: 4500 /unlink in 29.14 seconds: 154.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 748 (time 1787629580.03 total 10.00 last 74.79) - open/close 1570 (time 1787629590.04 total 20.01 last 82.09) - open/close 2477 (time 1787629600.06 total 30.03 last 90.57) - open/close 3279 (time 1787629610.06 total 40.03 last 80.16) - open/close 4104 (time 1787629620.06 total 50.03 last 82.50) total: 4500 open/close in 55.00 seconds: 81.81 ops/second - /unlink 930 (time 1787629641.04 total 10.00 last 93.00) - /unlink 1991 (time 1787629651.04 total 20.00 last 106.05) - /unlink 3136 (time 1787629661.05 total 30.01 last 114.43) total: 4500 /unlink in 39.33 seconds: 114.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 870 (time 1787629684.63 total 10.06 last 86.49) - open/close 1743 (time 1787629694.63 total 20.06 last 87.29) - open/close 2493 (time 1787629704.63 total 30.06 last 74.98) - open/close 3192 (time 1787629714.64 total 40.07 last 69.85) - open/close 3898 (time 1787629724.65 total 50.09 last 70.48) total: 4500 open/close in 57.57 seconds: 78.16 ops/second - /unlink 1674 (time 1787629745.65 total 10.01 last 167.25) - /unlink 3131 (time 1787629755.65 total 20.01 last 145.66) total: 4500 /unlink in 27.66 seconds: 162.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 784 (time 1787629776.39 total 10.01 last 78.35) - open/close 1566 (time 1787629786.39 total 20.01 last 78.18) - open/close 2285 (time 1787629796.41 total 30.02 last 71.78) - open/close 3128 (time 1787629806.42 total 40.04 last 84.18) - open/close 3929 (time 1787629816.43 total 50.05 last 80.02) total: 4500 open/close in 59.64 seconds: 75.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1182 (time 1787629839.86 total 10.01 last 118.13) - /unlink 2600 (time 1787629849.87 total 20.02 last 141.67) - /unlink 3841 (time 1787629859.87 total 30.02 last 124.10) total: 4500 /unlink in 34.85 seconds: 129.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 681 (time 1787629880.58 total 10.00 last 68.09) - open/close 1226 (time 1787629890.59 total 20.01 last 54.45) - open/close 1892 (time 1787629900.59 total 30.01 last 66.60) - open/close 2737 (time 1787629910.60 total 40.02 last 84.41) - open/close 3513 (time 1787629920.60 total 50.03 last 77.57) - open/close 4327 (time 1787629930.61 total 60.04 last 81.30) total: 4500 open/close in 62.41 seconds: 72.10 ops/second - /unlink 1301 (time 1787629948.99 total 10.00 last 130.10) - /unlink 2800 (time 1787629958.99 total 20.00 last 149.90) - /unlink 4163 (time 1787629968.99 total 30.00 last 136.25) total: 4500 /unlink in 32.99 seconds: 136.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 747 (time 1787629988.41 total 10.01 last 74.64) - open/close 1377 (time 1787629998.41 total 20.01 last 62.98) - open/close 2086 (time 1787630008.41 total 30.01 last 70.89) - open/close 2868 (time 1787630018.42 total 40.02 last 78.14) - open/close 3663 (time 1787630028.43 total 50.03 last 79.43) - open/close 4472 (time 1787630038.44 total 60.04 last 80.83) total: 4500 open/close in 60.46 seconds: 74.43 ops/second - /unlink 1279 (time 1787630053.29 total 10.00 last 127.88) - /unlink 2645 (time 1787630063.30 total 20.01 last 136.47) - /unlink 4131 (time 1787630073.31 total 30.02 last 148.47) total: 4500 /unlink in 32.79 seconds: 137.26 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95216 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local oleg311-server: modules unloaded.