== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:36:56 (1786804616) Stopping clients: oleg452-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg452-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg452-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg452-server oleg452-server: oleg452-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg452-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg452-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 675 (time 1786804816.51 total 10.01 last 67.45) - open/close 1309 (time 1786804826.52 total 20.02 last 63.33) - open/close 1947 (time 1786804836.53 total 30.02 last 63.76) - open/close 2598 (time 1786804846.53 total 40.03 last 65.08) - open/close 3298 (time 1786804856.54 total 50.04 last 69.92) - open/close 3969 (time 1786804866.55 total 60.04 last 67.07) total: 4500 open/close in 68.21 seconds: 65.98 ops/second - /unlink 1289 (time 1786804889.25 total 10.01 last 128.83) - /unlink 2517 (time 1786804899.26 total 20.01 last 122.74) - /unlink 3857 (time 1786804909.26 total 30.01 last 133.96) total: 4500 /unlink in 35.08 seconds: 128.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 751 (time 1786804928.96 total 10.01 last 75.05) - open/close 1425 (time 1786804938.96 total 20.01 last 67.36) - open/close 2067 (time 1786804948.98 total 30.03 last 64.08) - open/close 2768 (time 1786804958.98 total 40.03 last 70.08) - open/close 3359 (time 1786804968.99 total 50.04 last 59.07) - open/close 3964 (time 1786804979.03 total 60.08 last 60.27) total: 4500 open/close in 68.86 seconds: 65.35 ops/second - /unlink 1340 (time 1786805002.16 total 10.01 last 133.91) - /unlink 2620 (time 1786805012.17 total 20.01 last 127.92) - /unlink 3878 (time 1786805022.17 total 30.02 last 125.78) total: 4500 /unlink in 35.72 seconds: 125.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 634 (time 1786805043.58 total 10.00 last 63.37) - open/close 1284 (time 1786805053.59 total 20.02 last 64.93) - open/close 1997 (time 1786805063.60 total 30.03 last 71.18) - open/close 2755 (time 1786805073.62 total 40.05 last 75.71) - open/close 3426 (time 1786805083.62 total 50.05 last 67.08) - open/close 4141 (time 1786805093.63 total 60.06 last 71.40) total: 4500 open/close in 64.95 seconds: 69.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1157 (time 1786805112.85 total 10.00 last 115.66) - /unlink 2457 (time 1786805122.85 total 20.00 last 129.99) - /unlink 3619 (time 1786805132.86 total 30.01 last 116.17) total: 4500 /unlink in 37.28 seconds: 120.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 595 (time 1786805155.48 total 10.00 last 59.49) - open/close 1210 (time 1786805165.49 total 20.01 last 61.44) - open/close 1893 (time 1786805175.52 total 30.05 last 68.07) - open/close 2548 (time 1786805185.53 total 40.05 last 65.48) - open/close 3242 (time 1786805195.53 total 50.05 last 69.37) - open/close 4000 (time 1786805205.53 total 60.05 last 75.80) total: 4500 open/close in 68.08 seconds: 66.10 ops/second - /unlink 1086 (time 1786805228.18 total 10.02 last 108.43) - /unlink 1952 (time 1786805238.18 total 20.02 last 86.60) - /unlink 2809 (time 1786805248.18 total 30.02 last 85.64) - /unlink 3827 (time 1786805258.19 total 40.03 last 101.75) total: 4500 /unlink in 46.61 seconds: 96.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 594 (time 1786805282.14 total 10.01 last 59.34) - open/close 1181 (time 1786805292.15 total 20.01 last 58.67) - open/close 1764 (time 1786805302.15 total 30.02 last 58.28) - open/close 2369 (time 1786805312.16 total 40.03 last 60.45) - open/close 2971 (time 1786805322.17 total 50.03 last 60.17) - open/close 3669 (time 1786805332.18 total 60.05 last 69.68) - open/close 4337 (time 1786805342.22 total 70.08 last 66.59) total: 4500 open/close in 72.85 seconds: 61.77 ops/second - /unlink 1074 (time 1786805359.31 total 10.00 last 107.37) - /unlink 2173 (time 1786805369.32 total 20.01 last 109.86) - /unlink 3363 (time 1786805379.32 total 30.01 last 118.93) total: 4500 /unlink in 39.84 seconds: 112.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 627 (time 1786805403.71 total 10.00 last 62.70) - open/close 1332 (time 1786805413.72 total 20.01 last 70.46) - open/close 2044 (time 1786805423.72 total 30.01 last 71.16) - open/close 2807 (time 1786805433.73 total 40.02 last 76.26) - open/close 3495 (time 1786805443.74 total 50.03 last 68.69) - open/close 4327 (time 1786805453.74 total 60.03 last 83.20) total: 4500 open/close in 62.16 seconds: 72.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1331 (time 1786805470.09 total 10.00 last 133.05) - /unlink 2561 (time 1786805480.10 total 20.01 last 122.96) - /unlink 3999 (time 1786805490.10 total 30.01 last 143.73) total: 4500 /unlink in 33.36 seconds: 134.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 726 (time 1786805508.42 total 10.01 last 72.55) - open/close 1407 (time 1786805518.44 total 20.02 last 68.01) - open/close 2062 (time 1786805528.45 total 30.03 last 65.41) - open/close 2748 (time 1786805538.47 total 40.05 last 68.51) - open/close 3342 (time 1786805548.48 total 50.06 last 59.32) - open/close 4144 (time 1786805558.48 total 60.06 last 80.19) total: 4500 open/close in 65.40 seconds: 68.80 ops/second - /unlink 1088 (time 1786805578.53 total 10.00 last 108.78) - /unlink 2515 (time 1786805588.53 total 20.01 last 142.65) - /unlink 3522 (time 1786805598.54 total 30.01 last 100.65) - /unlink 4443 (time 1786805608.54 total 40.02 last 92.05) total: 4500 /unlink in 41.17 seconds: 109.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 610 (time 1786805625.93 total 10.01 last 60.95) - open/close 1283 (time 1786805635.94 total 20.02 last 67.20) - open/close 2084 (time 1786805645.95 total 30.03 last 80.02) - open/close 2764 (time 1786805655.95 total 40.03 last 68.00) - open/close 3479 (time 1786805665.95 total 50.04 last 71.49) - open/close 4185 (time 1786805675.96 total 60.05 last 70.52) total: 4500 open/close in 64.13 seconds: 70.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1254 (time 1786805695.21 total 10.01 last 125.28) - /unlink 2428 (time 1786805705.22 total 20.01 last 117.35) - /unlink 3458 (time 1786805715.25 total 30.05 last 102.63) - /unlink 4323 (time 1786805725.27 total 40.06 last 86.37) total: 4500 /unlink in 41.99 seconds: 107.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 589 (time 1786805742.41 total 10.00 last 58.89) - open/close 1229 (time 1786805752.42 total 20.01 last 63.96) - open/close 1792 (time 1786805762.43 total 30.02 last 56.24) - open/close 2411 (time 1786805772.44 total 40.03 last 61.83) - open/close 3192 (time 1786805782.45 total 50.04 last 78.01) - open/close 3993 (time 1786805792.46 total 60.05 last 80.04) total: 4500 open/close in 66.66 seconds: 67.50 ops/second - /unlink 1358 (time 1786805814.54 total 10.01 last 135.71) - /unlink 2740 (time 1786805824.55 total 20.01 last 138.09) - /unlink 4111 (time 1786805834.56 total 30.02 last 136.96) total: 4500 /unlink in 33.52 seconds: 134.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 727 (time 1786805852.91 total 10.01 last 72.64) - open/close 1276 (time 1786805862.92 total 20.01 last 54.87) - open/close 1778 (time 1786805872.95 total 30.04 last 50.06) - open/close 2471 (time 1786805882.96 total 40.05 last 69.23) - open/close 3091 (time 1786805892.96 total 50.06 last 61.98) - open/close 3796 (time 1786805902.97 total 60.07 last 70.43) total: 4500 open/close in 68.71 seconds: 65.50 ops/second - /unlink 1315 (time 1786805925.47 total 10.01 last 131.43) - /unlink 2590 (time 1786805935.48 total 20.01 last 127.46) - /unlink 4038 (time 1786805945.48 total 30.01 last 144.77) total: 4500 /unlink in 33.03 seconds: 136.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 708 (time 1786805962.57 total 10.01 last 70.74) - open/close 1404 (time 1786805972.58 total 20.01 last 69.56) - open/close 2094 (time 1786805982.58 total 30.02 last 68.99) - open/close 2844 (time 1786805992.59 total 40.02 last 74.94) - open/close 3581 (time 1786806002.60 total 50.04 last 73.58) - open/close 4276 (time 1786806012.62 total 60.06 last 69.39) total: 4500 open/close in 63.91 seconds: 70.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1122 (time 1786806031.98 total 10.01 last 112.14) - /unlink 2356 (time 1786806042.00 total 20.03 last 123.14) - /unlink 3675 (time 1786806052.01 total 30.03 last 131.83) total: 4500 /unlink in 37.26 seconds: 120.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 770 (time 1786806074.22 total 10.01 last 76.94) - open/close 1508 (time 1786806084.22 total 20.01 last 73.75) - open/close 2293 (time 1786806094.22 total 30.01 last 78.50) - open/close 3053 (time 1786806104.23 total 40.02 last 75.94) - open/close 3810 (time 1786806114.24 total 50.03 last 75.66) total: 4500 open/close in 59.79 seconds: 75.26 ops/second - /unlink 1363 (time 1786806138.92 total 10.00 last 136.28) - /unlink 2589 (time 1786806148.93 total 20.01 last 122.47) - /unlink 3873 (time 1786806158.94 total 30.01 last 128.36) total: 4500 /unlink in 36.61 seconds: 122.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 535 (time 1786806182.59 total 10.01 last 53.47) - open/close 1262 (time 1786806192.60 total 20.01 last 72.66) - open/close 1938 (time 1786806202.61 total 30.02 last 67.52) - open/close 2705 (time 1786806212.62 total 40.03 last 76.63) - open/close 3482 (time 1786806222.63 total 50.04 last 77.64) - open/close 4196 (time 1786806232.63 total 60.04 last 71.39) total: 4500 open/close in 65.12 seconds: 69.10 ops/second - /unlink 1304 (time 1786806252.44 total 10.01 last 130.32) - /unlink 2426 (time 1786806262.47 total 20.03 last 111.96) - /unlink 3660 (time 1786806272.47 total 30.03 last 123.40) total: 4500 /unlink in 36.16 seconds: 124.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 603 (time 1786806294.99 total 10.00 last 60.27) - open/close 1178 (time 1786806305.00 total 20.01 last 57.45) - open/close 1758 (time 1786806315.01 total 30.02 last 57.93) - open/close 2426 (time 1786806325.02 total 40.04 last 66.71) - open/close 3077 (time 1786806335.03 total 50.05 last 65.03) - open/close 3737 (time 1786806345.04 total 60.06 last 65.95) - open/close 4389 (time 1786806355.04 total 70.06 last 65.19) total: 4500 open/close in 71.37 seconds: 63.06 ops/second - /unlink 1179 (time 1786806370.90 total 10.01 last 117.84) - /unlink 2353 (time 1786806380.90 total 20.01 last 117.35) - /unlink 3468 (time 1786806390.90 total 30.01 last 111.50) total: 4500 /unlink in 39.20 seconds: 114.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 695 (time 1786806415.31 total 10.01 last 69.42) - open/close 1410 (time 1786806425.32 total 20.02 last 71.42) - open/close 2015 (time 1786806435.33 total 30.03 last 60.45) - open/close 2639 (time 1786806445.34 total 40.04 last 62.33) - open/close 3123 (time 1786806455.35 total 50.06 last 48.34) - open/close 3620 (time 1786806465.36 total 60.07 last 49.64) - open/close 4064 (time 1786806475.38 total 70.08 last 44.33) total: 4500 open/close in 78.64 seconds: 57.23 ops/second - /unlink 1268 (time 1786806498.86 total 10.00 last 126.78) - /unlink 2453 (time 1786806508.89 total 20.03 last 118.19) - /unlink 3755 (time 1786806518.89 total 30.03 last 130.17) total: 4500 /unlink in 35.39 seconds: 127.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 646 (time 1786806539.89 total 10.01 last 64.53) - open/close 1225 (time 1786806549.89 total 20.01 last 57.90) - open/close 1749 (time 1786806559.90 total 30.02 last 52.33) - open/close 2236 (time 1786806569.91 total 40.03 last 48.66) - open/close 2741 (time 1786806579.91 total 50.03 last 50.50) - open/close 3359 (time 1786806589.92 total 60.04 last 61.74) - open/close 4005 (time 1786806599.93 total 70.05 last 64.56) total: 4500 open/close in 77.08 seconds: 58.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1191 (time 1786806621.83 total 10.01 last 118.98) - /unlink 2555 (time 1786806631.85 total 20.03 last 136.17) - /unlink 3713 (time 1786806641.86 total 30.04 last 115.65) total: 4500 /unlink in 36.39 seconds: 123.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 514 (time 1786806664.05 total 10.01 last 51.32) - open/close 1182 (time 1786806674.06 total 20.02 last 66.75) - open/close 1929 (time 1786806684.06 total 30.02 last 74.70) - open/close 2691 (time 1786806694.06 total 40.03 last 76.18) - open/close 3349 (time 1786806704.08 total 50.04 last 65.67) - open/close 3994 (time 1786806714.08 total 60.05 last 64.48) total: 4500 open/close in 68.91 seconds: 65.31 ops/second - /unlink 1362 (time 1786806738.79 total 10.00 last 136.16) - /unlink 2400 (time 1786806748.82 total 20.03 last 103.48) - /unlink 3441 (time 1786806758.82 total 30.04 last 104.06) - /unlink 4399 (time 1786806768.83 total 40.04 last 95.77) total: 4500 /unlink in 41.68 seconds: 107.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 594 (time 1786806787.71 total 10.01 last 59.34) - open/close 1284 (time 1786806797.72 total 20.02 last 68.94) - open/close 1938 (time 1786806807.72 total 30.02 last 65.37) - open/close 2563 (time 1786806817.75 total 40.05 last 62.37) - open/close 3111 (time 1786806827.76 total 50.06 last 54.70) - open/close 3621 (time 1786806837.79 total 60.09 last 50.88) - open/close 4188 (time 1786806847.79 total 70.09 last 56.68) total: 4500 open/close in 74.21 seconds: 60.64 ops/second - /unlink 1300 (time 1786806866.99 total 10.00 last 130.00) - /unlink 2652 (time 1786806877.01 total 20.01 last 135.04) - /unlink 3876 (time 1786806887.02 total 30.02 last 122.25) total: 4500 /unlink in 35.21 seconds: 127.81 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11297: 95254 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server unloading modules via unload_modules_local on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local oleg452-server: modules unloaded.