== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:03:08 (1786831388) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 806 (time 1786831620.88 total 10.02 last 80.48) - open/close 1491 (time 1786831630.88 total 20.02 last 68.49) - open/close 2069 (time 1786831640.91 total 30.04 last 57.64) - open/close 2608 (time 1786831650.94 total 40.08 last 53.72) - open/close 3279 (time 1786831660.95 total 50.09 last 67.02) - open/close 3974 (time 1786831670.95 total 60.09 last 69.47) total: 4500 open/close in 68.92 seconds: 65.29 ops/second - /unlink 1314 (time 1786831695.44 total 10.02 last 131.20) - /unlink 2597 (time 1786831705.45 total 20.02 last 128.19) - /unlink 3961 (time 1786831715.46 total 30.03 last 136.32) total: 4500 /unlink in 33.98 seconds: 132.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 792 (time 1786831734.48 total 10.02 last 79.02) - open/close 1601 (time 1786831744.49 total 20.03 last 80.82) - open/close 2369 (time 1786831754.52 total 30.06 last 76.61) - open/close 2991 (time 1786831764.52 total 40.06 last 62.17) - open/close 3663 (time 1786831774.53 total 50.07 last 67.16) - open/close 4296 (time 1786831784.53 total 60.07 last 63.30) total: 4500 open/close in 62.75 seconds: 71.71 ops/second - /unlink 1222 (time 1786831802.45 total 10.00 last 122.20) - /unlink 2473 (time 1786831812.45 total 20.00 last 125.06) - /unlink 3782 (time 1786831822.46 total 30.01 last 130.81) total: 4500 /unlink in 36.41 seconds: 123.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 695 (time 1786831844.13 total 10.00 last 69.47) - open/close 1346 (time 1786831854.13 total 20.01 last 65.10) - open/close 2028 (time 1786831864.13 total 30.01 last 68.19) - open/close 2825 (time 1786831874.13 total 40.01 last 79.70) - open/close 3526 (time 1786831884.15 total 50.03 last 69.95) - open/close 4233 (time 1786831894.15 total 60.03 last 70.70) total: 4500 open/close in 63.71 seconds: 70.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1336 (time 1786831914.55 total 10.00 last 133.58) - /unlink 2342 (time 1786831924.55 total 20.01 last 100.56) - /unlink 3379 (time 1786831934.57 total 30.02 last 103.50) total: 4500 /unlink in 39.85 seconds: 112.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 665 (time 1786831960.51 total 10.00 last 66.49) - open/close 1322 (time 1786831970.54 total 20.03 last 65.50) - open/close 1981 (time 1786831980.55 total 30.04 last 65.87) - open/close 2694 (time 1786831990.57 total 40.06 last 71.14) - open/close 3413 (time 1786832000.58 total 50.07 last 71.87) - open/close 4133 (time 1786832010.58 total 60.07 last 71.95) total: 4500 open/close in 63.98 seconds: 70.34 ops/second - /unlink 1246 (time 1786832029.37 total 10.00 last 124.59) - /unlink 2557 (time 1786832039.37 total 20.00 last 131.07) - /unlink 3801 (time 1786832049.37 total 30.01 last 124.36) total: 4500 /unlink in 35.92 seconds: 125.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 641 (time 1786832071.75 total 10.01 last 64.05) - open/close 1276 (time 1786832081.76 total 20.02 last 63.42) - open/close 2001 (time 1786832091.77 total 30.03 last 72.42) - open/close 2701 (time 1786832101.78 total 40.04 last 69.94) - open/close 3344 (time 1786832111.79 total 50.05 last 64.23) - open/close 3992 (time 1786832121.80 total 60.06 last 64.75) total: 4500 open/close in 66.66 seconds: 67.51 ops/second - /unlink 1366 (time 1786832142.83 total 10.00 last 136.55) - /unlink 2874 (time 1786832152.83 total 20.01 last 150.74) - /unlink 4180 (time 1786832162.84 total 30.01 last 130.56) total: 4500 /unlink in 32.21 seconds: 139.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 754 (time 1786832180.43 total 10.02 last 75.26) - open/close 1585 (time 1786832190.44 total 20.02 last 83.06) - open/close 2356 (time 1786832200.44 total 30.02 last 77.09) - open/close 2847 (time 1786832210.45 total 40.04 last 49.04) - open/close 3397 (time 1786832220.46 total 50.05 last 54.92) - open/close 4014 (time 1786832230.47 total 60.05 last 61.68) total: 4500 open/close in 66.73 seconds: 67.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1296 (time 1786832251.74 total 10.02 last 129.40) - /unlink 2563 (time 1786832261.75 total 20.02 last 126.59) - /unlink 3533 (time 1786832271.75 total 30.03 last 96.92) total: 4500 /unlink in 39.36 seconds: 114.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 832 (time 1786832298.63 total 10.01 last 83.14) - open/close 1486 (time 1786832308.64 total 20.02 last 65.35) - open/close 2223 (time 1786832318.65 total 30.02 last 73.64) - open/close 2901 (time 1786832328.66 total 40.03 last 67.73) - open/close 3353 (time 1786832338.75 total 50.12 last 44.81) - open/close 3858 (time 1786832348.75 total 60.12 last 50.48) - open/close 4494 (time 1786832358.75 total 70.13 last 63.58) total: 4500 open/close in 70.16 seconds: 64.14 ops/second - /unlink 1463 (time 1786832375.27 total 10.00 last 146.24) - /unlink 2843 (time 1786832385.30 total 20.03 last 137.60) - /unlink 4212 (time 1786832395.30 total 30.04 last 136.86) total: 4500 /unlink in 32.36 seconds: 139.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 757 (time 1786832414.17 total 10.01 last 75.61) - open/close 1562 (time 1786832424.17 total 20.02 last 80.45) - open/close 2245 (time 1786832434.18 total 30.02 last 68.26) - open/close 3137 (time 1786832444.18 total 40.03 last 89.19) - open/close 3978 (time 1786832454.19 total 50.03 last 84.05) total: 4500 open/close in 57.38 seconds: 78.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1509 (time 1786832475.96 total 10.00 last 150.88) - /unlink 2690 (time 1786832485.97 total 20.01 last 118.03) - /unlink 4191 (time 1786832495.97 total 30.01 last 150.07) total: 4500 /unlink in 32.14 seconds: 140.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 754 (time 1786832512.39 total 10.00 last 75.37) - open/close 1386 (time 1786832522.39 total 20.00 last 63.20) - open/close 2006 (time 1786832532.39 total 30.01 last 61.98) - open/close 2657 (time 1786832542.39 total 40.01 last 65.07) - open/close 3335 (time 1786832552.42 total 50.04 last 67.64) - open/close 4140 (time 1786832562.43 total 60.05 last 80.40) total: 4500 open/close in 64.75 seconds: 69.50 ops/second - /unlink 1363 (time 1786832582.42 total 10.01 last 136.15) - /unlink 2971 (time 1786832592.42 total 20.02 last 160.74) - /unlink 4402 (time 1786832602.43 total 30.02 last 142.97) total: 4500 /unlink in 30.92 seconds: 145.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 745 (time 1786832618.82 total 10.01 last 74.42) - open/close 1433 (time 1786832628.84 total 20.03 last 68.70) - open/close 2069 (time 1786832638.86 total 30.04 last 63.48) - open/close 2633 (time 1786832648.86 total 40.05 last 56.36) - open/close 3316 (time 1786832658.86 total 50.05 last 68.29) - open/close 3920 (time 1786832668.87 total 60.06 last 60.38) total: 4500 open/close in 68.47 seconds: 65.73 ops/second - /unlink 1428 (time 1786832692.87 total 10.05 last 142.12) - /unlink 2687 (time 1786832702.87 total 20.05 last 125.89) - /unlink 3888 (time 1786832712.87 total 30.05 last 120.09) total: 4500 /unlink in 36.19 seconds: 124.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 825 (time 1786832734.76 total 10.00 last 82.49) - open/close 1532 (time 1786832744.77 total 20.01 last 70.64) - open/close 2242 (time 1786832754.77 total 30.02 last 70.97) - open/close 3100 (time 1786832764.78 total 40.02 last 85.73) - open/close 3949 (time 1786832774.78 total 50.02 last 84.90) total: 4500 open/close in 57.07 seconds: 78.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1388 (time 1786832796.11 total 10.01 last 138.67) - /unlink 2779 (time 1786832806.12 total 20.02 last 139.02) - /unlink 4117 (time 1786832816.12 total 30.02 last 133.76) total: 4500 /unlink in 32.46 seconds: 138.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 885 (time 1786832833.70 total 10.01 last 88.43) - open/close 1674 (time 1786832843.72 total 20.03 last 78.74) - open/close 2468 (time 1786832853.72 total 30.04 last 79.34) - open/close 3198 (time 1786832863.73 total 40.04 last 72.95) - open/close 4032 (time 1786832873.74 total 50.05 last 83.34) total: 4500 open/close in 56.12 seconds: 80.18 ops/second - /unlink 1429 (time 1786832894.28 total 10.00 last 142.87) - /unlink 2828 (time 1786832904.28 total 20.01 last 139.84) - /unlink 3981 (time 1786832914.29 total 30.01 last 115.23) total: 4500 /unlink in 34.78 seconds: 129.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 696 (time 1786832934.42 total 10.00 last 69.57) - open/close 1270 (time 1786832944.43 total 20.01 last 57.36) - open/close 1756 (time 1786832954.44 total 30.02 last 48.57) - open/close 2388 (time 1786832964.44 total 40.02 last 63.16) - open/close 2896 (time 1786832974.45 total 50.03 last 50.78) - open/close 3663 (time 1786832984.45 total 60.03 last 76.67) - open/close 4417 (time 1786832994.46 total 70.04 last 75.34) total: 4500 open/close in 71.68 seconds: 62.78 ops/second - /unlink 1067 (time 1786833011.96 total 10.00 last 106.66) - /unlink 2028 (time 1786833021.97 total 20.01 last 96.06) - /unlink 3024 (time 1786833031.97 total 30.01 last 99.58) - /unlink 3950 (time 1786833041.97 total 40.01 last 92.56) total: 4500 /unlink in 44.50 seconds: 101.11 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 573 (time 1786833063.25 total 10.02 last 57.19) - open/close 1046 (time 1786833073.26 total 20.03 last 47.26) - open/close 1680 (time 1786833083.26 total 30.03 last 63.36) - open/close 2226 (time 1786833093.28 total 40.05 last 54.51) - open/close 2901 (time 1786833103.29 total 50.06 last 67.43) - open/close 3710 (time 1786833113.29 total 60.06 last 80.89) - open/close 4369 (time 1786833123.31 total 70.08 last 65.80) total: 4500 open/close in 71.62 seconds: 62.83 ops/second - /unlink 1275 (time 1786833140.22 total 10.00 last 127.46) - /unlink 2399 (time 1786833150.23 total 20.01 last 112.36) - /unlink 3618 (time 1786833160.23 total 30.01 last 121.84) total: 4500 /unlink in 37.59 seconds: 119.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 717 (time 1786833183.90 total 10.01 last 71.63) - open/close 1423 (time 1786833193.91 total 20.02 last 70.56) - open/close 2000 (time 1786833203.93 total 30.03 last 57.61) - open/close 2601 (time 1786833213.93 total 40.03 last 60.10) - open/close 3287 (time 1786833223.93 total 50.03 last 68.58) - open/close 3930 (time 1786833233.95 total 60.05 last 64.17) total: 4500 open/close in 68.16 seconds: 66.02 ops/second - /unlink 1059 (time 1786833256.83 total 10.01 last 105.78) - /unlink 2062 (time 1786833266.84 total 20.02 last 100.26) - /unlink 3322 (time 1786833276.84 total 30.02 last 125.97) - /unlink 4475 (time 1786833286.84 total 40.02 last 115.27) total: 4500 /unlink in 40.20 seconds: 111.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 749 (time 1786833302.03 total 10.03 last 74.69) - open/close 1454 (time 1786833312.03 total 20.03 last 70.46) - open/close 2127 (time 1786833322.04 total 30.04 last 67.26) - open/close 2827 (time 1786833332.04 total 40.04 last 69.97) - open/close 3577 (time 1786833342.05 total 50.05 last 74.95) - open/close 4269 (time 1786833352.06 total 60.06 last 69.11) total: 4500 open/close in 64.10 seconds: 70.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1108 (time 1786833371.99 total 10.01 last 110.66) - /unlink 2329 (time 1786833381.99 total 20.01 last 122.09) - /unlink 3621 (time 1786833392.00 total 30.02 last 129.15) total: 4500 /unlink in 36.54 seconds: 123.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 582 (time 1786833414.46 total 10.02 last 58.07) - open/close 1191 (time 1786833424.47 total 20.03 last 60.86) - open/close 1864 (time 1786833434.48 total 30.03 last 67.26) - open/close 2536 (time 1786833444.51 total 40.07 last 66.98) - open/close 3173 (time 1786833454.52 total 50.07 last 63.65) - open/close 3827 (time 1786833464.54 total 60.10 last 65.26) total: 4500 open/close in 69.80 seconds: 64.47 ops/second - /unlink 1150 (time 1786833489.85 total 10.01 last 114.86) - /unlink 2303 (time 1786833499.85 total 20.01 last 115.29) - /unlink 3628 (time 1786833509.86 total 30.02 last 132.39) total: 4500 /unlink in 37.66 seconds: 119.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 681 (time 1786833532.93 total 10.01 last 68.06) - open/close 1253 (time 1786833542.93 total 20.01 last 57.19) - open/close 1828 (time 1786833552.95 total 30.02 last 57.42) - open/close 2402 (time 1786833562.96 total 40.04 last 57.31) - open/close 3176 (time 1786833572.98 total 50.05 last 77.27) - open/close 3985 (time 1786833582.99 total 60.06 last 80.84) total: 4500 open/close in 68.03 seconds: 66.14 ops/second - /unlink 1320 (time 1786833606.19 total 10.02 last 131.79) - /unlink 2729 (time 1786833616.20 total 20.02 last 140.87) - /unlink 4035 (time 1786833626.20 total 30.02 last 130.57) total: 4500 /unlink in 33.66 seconds: 133.71 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95344 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded.