== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:31:47 (1761240707) Stopping clients: oleg620-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg620-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg620-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg620-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg620-server oleg620-server: oleg620-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing load_modules_local oleg620-server: Loading modules from /home/green/git/lustre-release/lustre oleg620-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg620-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg620-server: detected 4 online CPUs by sysfs oleg620-server: MODOPTS_LIBCFS= oleg620-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg620-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg620-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg620-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg620-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg620-server: oleg620-server.virtnet: executing set_default_debug -1 all pdsh@oleg620-client: oleg620-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg620-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg620-client.virtnet: -o user_xattr,flock 192.168.206.120@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 875 (time 1761240878.47 total 10.01 last 87.43) - open/close 1735 (time 1761240888.48 total 20.02 last 85.91) - open/close 2547 (time 1761240898.49 total 30.03 last 81.12) - open/close 3380 (time 1761240908.50 total 40.04 last 83.21) - open/close 4096 (time 1761240918.50 total 50.04 last 71.59) total: 4500 open/close in 55.14 seconds: 81.61 ops/second - /unlink 1573 (time 1761240937.56 total 10.00 last 157.25) - /unlink 3052 (time 1761240947.56 total 20.01 last 147.80) - /unlink 4310 (time 1761240957.57 total 30.01 last 125.78) total: 4500 /unlink in 31.80 seconds: 141.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 673 (time 1761240974.31 total 10.00 last 67.30) - open/close 1438 (time 1761240984.32 total 20.01 last 76.44) - open/close 2301 (time 1761240994.33 total 30.02 last 86.21) - open/close 3058 (time 1761241004.34 total 40.03 last 75.61) - open/close 3885 (time 1761241014.35 total 50.04 last 82.63) total: 4500 open/close in 57.19 seconds: 78.69 ops/second - /unlink 1599 (time 1761241035.12 total 10.00 last 159.86) - /unlink 3134 (time 1761241045.13 total 20.01 last 153.44) total: 4500 /unlink in 29.09 seconds: 154.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 827 (time 1761241068.69 total 10.01 last 82.66) - open/close 1662 (time 1761241078.71 total 20.02 last 83.35) - open/close 2510 (time 1761241088.71 total 30.03 last 84.75) - open/close 3330 (time 1761241098.72 total 40.03 last 81.97) - open/close 4133 (time 1761241108.73 total 50.04 last 80.22) total: 4500 open/close in 54.56 seconds: 82.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1630 (time 1761241127.16 total 10.00 last 162.93) - /unlink 3192 (time 1761241137.16 total 20.01 last 156.09) total: 4500 /unlink in 28.38 seconds: 158.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 864 (time 1761241160.10 total 10.01 last 86.30) - open/close 1769 (time 1761241170.10 total 20.01 last 90.50) - open/close 2669 (time 1761241180.11 total 30.02 last 89.95) - open/close 3635 (time 1761241190.12 total 40.03 last 96.50) total: 4500 open/close in 49.35 seconds: 91.18 ops/second - /unlink 1667 (time 1761241212.45 total 10.00 last 166.68) - /unlink 3424 (time 1761241222.46 total 20.01 last 175.48) total: 4500 /unlink in 26.47 seconds: 169.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 890 (time 1761241242.08 total 10.01 last 88.93) - open/close 1792 (time 1761241252.08 total 20.02 last 90.11) - open/close 2669 (time 1761241262.09 total 30.02 last 87.64) - open/close 3538 (time 1761241272.10 total 40.03 last 86.87) - open/close 4437 (time 1761241282.10 total 50.03 last 89.90) total: 4500 open/close in 50.60 seconds: 88.94 ops/second - /unlink 1605 (time 1761241296.12 total 10.00 last 160.47) - /unlink 3124 (time 1761241306.12 total 20.01 last 151.84) total: 4500 /unlink in 28.71 seconds: 156.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 849 (time 1761241328.90 total 10.01 last 84.77) - open/close 1618 (time 1761241338.91 total 20.03 last 76.82) - open/close 2530 (time 1761241348.92 total 30.03 last 91.13) - open/close 3396 (time 1761241358.93 total 40.04 last 86.56) - open/close 4325 (time 1761241368.93 total 50.04 last 92.87) total: 4500 open/close in 52.03 seconds: 86.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1600 (time 1761241383.87 total 10.01 last 159.92) - /unlink 3292 (time 1761241393.87 total 20.01 last 169.13) total: 4500 /unlink in 26.88 seconds: 167.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 959 (time 1761241413.66 total 10.00 last 95.88) - open/close 1867 (time 1761241423.66 total 20.00 last 90.80) - open/close 2823 (time 1761241433.68 total 30.02 last 95.48) - open/close 3750 (time 1761241443.68 total 40.02 last 92.66) total: 4500 open/close in 47.71 seconds: 94.32 ops/second - /unlink 1658 (time 1761241464.41 total 10.01 last 165.64) - /unlink 3266 (time 1761241474.42 total 20.01 last 160.72) total: 4500 /unlink in 27.59 seconds: 163.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 896 (time 1761241495.66 total 10.01 last 89.53) - open/close 1762 (time 1761241505.67 total 20.01 last 86.54) - open/close 2628 (time 1761241515.68 total 30.03 last 86.49) - open/close 3500 (time 1761241525.69 total 40.04 last 87.12) - open/close 4443 (time 1761241535.70 total 50.04 last 94.23) total: 4500 open/close in 50.56 seconds: 89.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1876 (time 1761241548.42 total 10.00 last 187.53) - /unlink 3401 (time 1761241558.42 total 20.01 last 152.47) total: 4500 /unlink in 26.57 seconds: 169.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 812 (time 1761241578.75 total 10.01 last 81.10) - open/close 1458 (time 1761241588.76 total 20.02 last 64.57) - open/close 2175 (time 1761241598.76 total 30.03 last 71.64) - open/close 3069 (time 1761241608.76 total 40.03 last 89.40) - open/close 3946 (time 1761241618.78 total 50.04 last 87.55) total: 4500 open/close in 56.22 seconds: 80.05 ops/second - /unlink 1708 (time 1761241638.56 total 10.00 last 170.77) - /unlink 3200 (time 1761241648.56 total 20.01 last 149.08) total: 4500 /unlink in 28.23 seconds: 159.39 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 922 (time 1761241670.12 total 10.00 last 92.16) - open/close 1779 (time 1761241680.12 total 20.01 last 85.69) - open/close 2727 (time 1761241690.13 total 30.01 last 94.74) - open/close 3614 (time 1761241700.13 total 40.02 last 88.66) - open/close 4414 (time 1761241710.13 total 50.02 last 79.98) total: 4500 open/close in 51.16 seconds: 87.96 ops/second - /unlink 1526 (time 1761241725.16 total 10.01 last 152.49) - /unlink 3097 (time 1761241735.17 total 20.02 last 156.97) total: 4500 /unlink in 28.46 seconds: 158.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 813 (time 1761241757.91 total 10.00 last 81.29) - open/close 1680 (time 1761241767.91 total 20.01 last 86.67) - open/close 2567 (time 1761241777.91 total 30.01 last 88.68) - open/close 3375 (time 1761241787.92 total 40.01 last 80.77) - open/close 4262 (time 1761241797.92 total 50.01 last 88.69) total: 4500 open/close in 52.74 seconds: 85.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1692 (time 1761241813.71 total 10.00 last 169.16) - /unlink 3261 (time 1761241823.72 total 20.01 last 156.84) total: 4500 /unlink in 27.65 seconds: 162.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 817 (time 1761241845.27 total 10.00 last 81.69) - open/close 1637 (time 1761241855.27 total 20.01 last 81.95) - open/close 2467 (time 1761241865.28 total 30.01 last 82.97) - open/close 3289 (time 1761241875.28 total 40.02 last 82.15) - open/close 4148 (time 1761241885.28 total 50.02 last 85.89) total: 4500 open/close in 54.22 seconds: 82.99 ops/second - /unlink 1634 (time 1761241903.82 total 10.01 last 163.27) - /unlink 3197 (time 1761241913.82 total 20.01 last 156.24) total: 4500 /unlink in 28.27 seconds: 159.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 803 (time 1761241935.62 total 10.00 last 80.28) - open/close 1593 (time 1761241945.63 total 20.01 last 78.93) - open/close 2251 (time 1761241955.63 total 30.01 last 65.78) - open/close 3013 (time 1761241965.64 total 40.02 last 76.14) - open/close 3770 (time 1761241975.66 total 50.04 last 75.59) total: 4500 open/close in 59.26 seconds: 75.94 ops/second - /unlink 1484 (time 1761241999.07 total 10.00 last 148.38) - /unlink 2829 (time 1761242009.08 total 20.01 last 134.41) - /unlink 4388 (time 1761242019.09 total 30.02 last 155.73) total: 4500 /unlink in 30.79 seconds: 146.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 750 (time 1761242033.77 total 10.01 last 74.95) - open/close 1569 (time 1761242043.77 total 20.01 last 81.88) - open/close 2356 (time 1761242053.78 total 30.02 last 78.65) - open/close 3188 (time 1761242063.78 total 40.02 last 83.18) - open/close 4049 (time 1761242073.79 total 50.03 last 86.03) total: 4500 open/close in 56.31 seconds: 79.92 ops/second - /unlink 1493 (time 1761242093.78 total 10.00 last 149.30) - /unlink 3011 (time 1761242103.79 total 20.00 last 151.73) - /unlink 4457 (time 1761242113.79 total 30.01 last 144.56) total: 4500 /unlink in 30.26 seconds: 148.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 878 (time 1761242127.88 total 10.01 last 87.71) - open/close 1669 (time 1761242137.88 total 20.01 last 79.08) - open/close 2485 (time 1761242147.88 total 30.01 last 81.59) - open/close 3332 (time 1761242157.88 total 40.02 last 84.67) - open/close 4169 (time 1761242167.89 total 50.02 last 83.65) total: 4500 open/close in 53.68 seconds: 83.83 ops/second - /unlink 1610 (time 1761242184.63 total 10.00 last 160.94) - /unlink 3174 (time 1761242194.63 total 20.01 last 156.37) total: 4500 /unlink in 28.86 seconds: 155.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 813 (time 1761242218.60 total 10.00 last 81.28) - open/close 1652 (time 1761242228.61 total 20.01 last 83.84) - open/close 2544 (time 1761242238.61 total 30.01 last 89.15) - open/close 3470 (time 1761242248.62 total 40.02 last 92.51) - open/close 4333 (time 1761242258.64 total 50.04 last 86.18) total: 4500 open/close in 51.98 seconds: 86.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1491 (time 1761242274.39 total 10.00 last 149.06) - /unlink 2897 (time 1761242284.39 total 20.01 last 140.55) - /unlink 4318 (time 1761242294.39 total 30.01 last 142.08) total: 4500 /unlink in 31.56 seconds: 142.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 754 (time 1761242310.29 total 10.01 last 75.32) - open/close 1577 (time 1761242320.29 total 20.02 last 82.26) - open/close 2397 (time 1761242330.30 total 30.02 last 81.93) - open/close 3196 (time 1761242340.31 total 40.03 last 79.84) - open/close 4023 (time 1761242350.31 total 50.04 last 82.65) total: 4500 open/close in 56.27 seconds: 79.97 ops/second - /unlink 1473 (time 1761242369.94 total 10.00 last 147.25) - /unlink 2867 (time 1761242379.94 total 20.01 last 139.37) - /unlink 4388 (time 1761242389.94 total 30.01 last 152.07) total: 4500 /unlink in 30.79 seconds: 146.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 882 (time 1761242404.18 total 10.00 last 88.16) - open/close 1675 (time 1761242414.18 total 20.01 last 79.28) - open/close 2508 (time 1761242424.19 total 30.01 last 83.23) - open/close 3251 (time 1761242434.20 total 40.02 last 74.23) - open/close 4068 (time 1761242444.21 total 50.03 last 81.62) total: 4500 open/close in 55.64 seconds: 80.88 ops/second - /unlink 1427 (time 1761242464.04 total 10.01 last 142.54) - /unlink 2961 (time 1761242474.04 total 20.02 last 153.33) - /unlink 4461 (time 1761242484.05 total 30.02 last 149.86) total: 4500 /unlink in 30.44 seconds: 147.84 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11102: 93713 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg620-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg620-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg620-server stop mds service on oleg620-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg620-server stop mds service on oleg620-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg620-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg620-server' oleg620-server: oleg620-server.virtnet: executing unload_modules_local oleg620-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg620-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.