== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:59:33 (1787597973) Stopping clients: oleg259-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg259-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg259-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg259-server oleg259-server: oleg259-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg259-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg259-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 712 (time 1787598182.97 total 10.00 last 71.17) - open/close 1391 (time 1787598193.01 total 20.05 last 67.57) - open/close 2097 (time 1787598203.02 total 30.06 last 70.55) - open/close 2825 (time 1787598213.04 total 40.08 last 72.63) - open/close 3454 (time 1787598223.06 total 50.10 last 62.78) - open/close 4169 (time 1787598233.07 total 60.11 last 71.44) total: 4500 open/close in 64.57 seconds: 69.70 ops/second - /unlink 1357 (time 1787598252.18 total 10.00 last 135.69) - /unlink 2927 (time 1787598262.18 total 20.00 last 157.00) - /unlink 4294 (time 1787598272.18 total 30.00 last 136.69) total: 4500 /unlink in 32.06 seconds: 140.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 891 (time 1787598289.04 total 10.01 last 89.03) - open/close 1633 (time 1787598299.05 total 20.03 last 74.07) - open/close 2299 (time 1787598309.06 total 30.03 last 66.54) - open/close 2832 (time 1787598319.06 total 40.03 last 53.30) - open/close 3557 (time 1787598329.07 total 50.04 last 72.48) - open/close 4282 (time 1787598339.07 total 60.04 last 72.45) total: 4500 open/close in 63.62 seconds: 70.73 ops/second - /unlink 1358 (time 1787598357.44 total 10.00 last 135.77) - /unlink 2656 (time 1787598367.44 total 20.01 last 129.74) - /unlink 3972 (time 1787598377.45 total 30.01 last 131.54) total: 4500 /unlink in 33.91 seconds: 132.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 819 (time 1787598396.57 total 10.00 last 81.87) - open/close 1644 (time 1787598406.58 total 20.01 last 82.44) - open/close 2330 (time 1787598416.58 total 30.02 last 68.57) - open/close 2973 (time 1787598426.59 total 40.03 last 64.23) - open/close 3712 (time 1787598436.60 total 50.03 last 73.86) total: 4500 open/close in 59.75 seconds: 75.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1219 (time 1787598461.74 total 10.01 last 121.82) - /unlink 2558 (time 1787598471.75 total 20.01 last 133.84) - /unlink 3767 (time 1787598481.75 total 30.01 last 120.86) total: 4500 /unlink in 36.31 seconds: 123.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 707 (time 1787598502.78 total 10.01 last 70.60) - open/close 1375 (time 1787598512.79 total 20.03 last 66.71) - open/close 2098 (time 1787598522.80 total 30.03 last 72.25) - open/close 2738 (time 1787598532.80 total 40.03 last 64.00) - open/close 3469 (time 1787598542.80 total 50.04 last 73.07) - open/close 4212 (time 1787598552.80 total 60.04 last 74.30) total: 4500 open/close in 63.53 seconds: 70.84 ops/second - /unlink 1481 (time 1787598570.71 total 10.00 last 148.07) - /unlink 2825 (time 1787598580.73 total 20.02 last 134.11) - /unlink 4028 (time 1787598590.74 total 30.03 last 120.20) total: 4500 /unlink in 33.68 seconds: 133.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 736 (time 1787598609.89 total 10.02 last 73.43) - open/close 1490 (time 1787598619.89 total 20.03 last 75.37) - open/close 2174 (time 1787598629.90 total 30.03 last 68.39) - open/close 2898 (time 1787598639.90 total 40.03 last 72.38) - open/close 3636 (time 1787598649.91 total 50.04 last 73.74) - open/close 4315 (time 1787598659.91 total 60.05 last 67.84) total: 4500 open/close in 63.09 seconds: 71.33 ops/second - /unlink 1176 (time 1787598678.82 total 10.00 last 117.60) - /unlink 2310 (time 1787598688.83 total 20.01 last 113.34) - /unlink 3693 (time 1787598698.83 total 30.01 last 138.24) total: 4500 /unlink in 36.58 seconds: 123.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 695 (time 1787598720.35 total 10.00 last 69.47) - open/close 1468 (time 1787598730.36 total 20.02 last 77.18) - open/close 2108 (time 1787598740.37 total 30.03 last 63.94) - open/close 2770 (time 1787598750.39 total 40.05 last 66.08) - open/close 3446 (time 1787598760.39 total 50.05 last 67.59) - open/close 4213 (time 1787598770.39 total 60.05 last 76.66) total: 4500 open/close in 63.80 seconds: 70.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1658 (time 1787598788.56 total 10.00 last 165.72) - /unlink 3068 (time 1787598798.58 total 20.02 last 140.80) - /unlink 4287 (time 1787598808.58 total 30.02 last 121.84) total: 4500 /unlink in 31.75 seconds: 141.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 647 (time 1787598827.31 total 10.01 last 64.67) - open/close 1317 (time 1787598837.32 total 20.01 last 66.96) - open/close 1968 (time 1787598847.32 total 30.02 last 65.07) - open/close 2542 (time 1787598857.33 total 40.02 last 57.35) - open/close 3155 (time 1787598867.34 total 50.03 last 61.26) - open/close 3761 (time 1787598877.36 total 60.05 last 60.47) - open/close 4497 (time 1787598887.36 total 70.06 last 73.55) total: 4500 open/close in 70.08 seconds: 64.21 ops/second - /unlink 1611 (time 1787598902.36 total 10.00 last 161.05) - /unlink 2952 (time 1787598912.36 total 20.01 last 134.04) - /unlink 4307 (time 1787598922.37 total 30.01 last 135.45) total: 4500 /unlink in 31.24 seconds: 144.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 738 (time 1787598937.99 total 10.00 last 73.78) - open/close 1449 (time 1787598947.99 total 20.00 last 71.09) - open/close 2090 (time 1787598958.00 total 30.01 last 64.05) - open/close 2730 (time 1787598968.01 total 40.02 last 63.92) - open/close 3434 (time 1787598978.02 total 50.03 last 70.34) - open/close 4122 (time 1787598988.03 total 60.04 last 68.75) total: 4500 open/close in 65.93 seconds: 68.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1276 (time 1787599008.48 total 10.00 last 127.55) - /unlink 2923 (time 1787599018.48 total 20.00 last 164.69) - /unlink 4317 (time 1787599028.48 total 30.00 last 139.40) total: 4500 /unlink in 31.59 seconds: 142.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 762 (time 1787599044.81 total 10.00 last 76.18) - open/close 1394 (time 1787599054.82 total 20.01 last 63.15) - open/close 2092 (time 1787599064.83 total 30.02 last 69.71) - open/close 2850 (time 1787599074.83 total 40.03 last 75.76) - open/close 3637 (time 1787599084.84 total 50.03 last 78.69) - open/close 4399 (time 1787599094.84 total 60.03 last 76.17) total: 4500 open/close in 61.35 seconds: 73.35 ops/second - /unlink 1218 (time 1787599110.99 total 10.00 last 121.78) - /unlink 2653 (time 1787599121.00 total 20.01 last 143.42) total: 4500 /unlink in 29.60 seconds: 152.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 836 (time 1787599144.59 total 10.00 last 83.58) - open/close 1510 (time 1787599154.61 total 20.02 last 67.27) - open/close 2261 (time 1787599164.61 total 30.02 last 75.08) - open/close 3071 (time 1787599174.62 total 40.03 last 80.95) - open/close 3906 (time 1787599184.62 total 50.03 last 83.47) total: 4500 open/close in 59.35 seconds: 75.82 ops/second - /unlink 1214 (time 1787599208.94 total 10.00 last 121.35) - /unlink 2392 (time 1787599218.94 total 20.01 last 117.74) - /unlink 3563 (time 1787599228.95 total 30.02 last 117.01) total: 4500 /unlink in 37.34 seconds: 120.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 680 (time 1787599252.65 total 10.00 last 67.98) - open/close 1371 (time 1787599262.65 total 20.01 last 69.06) - open/close 2181 (time 1787599272.66 total 30.02 last 80.94) - open/close 2841 (time 1787599282.68 total 40.04 last 65.85) - open/close 3420 (time 1787599292.71 total 50.06 last 57.77) - open/close 3890 (time 1787599302.71 total 60.07 last 46.96) - open/close 4455 (time 1787599312.75 total 70.10 last 56.32) total: 4500 open/close in 70.87 seconds: 63.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1525 (time 1787599328.78 total 10.01 last 152.37) - /unlink 2957 (time 1787599338.78 total 20.01 last 143.16) - /unlink 4346 (time 1787599348.79 total 30.02 last 138.78) total: 4500 /unlink in 31.48 seconds: 142.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 653 (time 1787599365.75 total 10.02 last 65.20) - open/close 1324 (time 1787599375.75 total 20.03 last 67.04) - open/close 1957 (time 1787599385.77 total 30.04 last 63.23) - open/close 2587 (time 1787599395.78 total 40.05 last 62.94) - open/close 3282 (time 1787599405.79 total 50.06 last 69.42) - open/close 3966 (time 1787599415.79 total 60.06 last 68.36) total: 4500 open/close in 66.70 seconds: 67.47 ops/second - /unlink 1348 (time 1787599437.13 total 10.00 last 134.74) - /unlink 2615 (time 1787599447.14 total 20.02 last 126.52) - /unlink 3798 (time 1787599457.15 total 30.03 last 118.20) total: 4500 /unlink in 35.72 seconds: 126.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 762 (time 1787599478.06 total 10.01 last 76.15) - open/close 1371 (time 1787599488.06 total 20.01 last 60.90) - open/close 2150 (time 1787599498.07 total 30.01 last 77.89) - open/close 2829 (time 1787599508.07 total 40.01 last 67.87) - open/close 3639 (time 1787599518.07 total 50.01 last 80.99) - open/close 4454 (time 1787599528.07 total 60.01 last 81.49) total: 4500 open/close in 60.80 seconds: 74.01 ops/second - /unlink 1318 (time 1787599543.82 total 10.00 last 131.78) - /unlink 2597 (time 1787599553.82 total 20.00 last 127.88) - /unlink 3919 (time 1787599563.83 total 30.01 last 132.14) total: 4500 /unlink in 34.53 seconds: 130.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 584 (time 1787599583.93 total 10.01 last 58.36) - open/close 1079 (time 1787599593.94 total 20.01 last 49.47) - open/close 1582 (time 1787599603.95 total 30.02 last 50.25) - open/close 2232 (time 1787599613.97 total 40.04 last 64.87) - open/close 2948 (time 1787599623.98 total 50.05 last 71.55) - open/close 3659 (time 1787599633.98 total 60.05 last 71.07) - open/close 4282 (time 1787599643.99 total 70.07 last 62.21) total: 4500 open/close in 73.62 seconds: 61.12 ops/second - /unlink 1246 (time 1787599663.97 total 10.01 last 124.44) - /unlink 2285 (time 1787599673.97 total 20.02 last 103.82) - /unlink 3604 (time 1787599683.98 total 30.03 last 131.75) total: 4500 /unlink in 36.34 seconds: 123.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 660 (time 1787599705.54 total 10.02 last 65.86) - open/close 1261 (time 1787599715.55 total 20.03 last 60.05) - open/close 1862 (time 1787599725.55 total 30.03 last 60.07) - open/close 2473 (time 1787599735.56 total 40.04 last 61.04) - open/close 3101 (time 1787599745.56 total 50.05 last 62.79) - open/close 3721 (time 1787599755.57 total 60.05 last 61.95) - open/close 4362 (time 1787599765.58 total 70.06 last 64.07) total: 4500 open/close in 71.96 seconds: 62.53 ops/second - /unlink 1107 (time 1787599782.39 total 10.00 last 110.65) - /unlink 2388 (time 1787599792.40 total 20.01 last 128.04) - /unlink 3547 (time 1787599802.40 total 30.01 last 115.87) total: 4500 /unlink in 37.25 seconds: 120.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 487 (time 1787599825.50 total 10.01 last 48.64) - open/close 1026 (time 1787599835.52 total 20.03 last 53.81) - open/close 1467 (time 1787599845.53 total 30.04 last 44.05) - open/close 2051 (time 1787599855.54 total 40.05 last 58.34) - open/close 2709 (time 1787599865.55 total 50.06 last 65.77) - open/close 3376 (time 1787599875.57 total 60.08 last 66.54) - open/close 3981 (time 1787599885.57 total 70.08 last 60.49) total: 4500 open/close in 77.87 seconds: 57.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1202 (time 1787599908.87 total 10.01 last 120.14) - /unlink 2500 (time 1787599918.87 total 20.01 last 129.80) - /unlink 3793 (time 1787599928.87 total 30.01 last 129.29) total: 4500 /unlink in 36.25 seconds: 124.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 573 (time 1787599951.42 total 10.02 last 57.16) - open/close 1088 (time 1787599961.44 total 20.04 last 51.40) - open/close 1773 (time 1787599971.45 total 30.05 last 68.46) - open/close 2496 (time 1787599981.45 total 40.06 last 72.26) - open/close 3181 (time 1787599991.46 total 50.06 last 68.49) - open/close 3728 (time 1787600001.47 total 60.07 last 54.64) - open/close 4165 (time 1787600011.47 total 70.07 last 43.69) total: 4500 open/close in 74.27 seconds: 60.59 ops/second - /unlink 1294 (time 1787600031.73 total 10.00 last 129.39) - /unlink 2646 (time 1787600041.73 total 20.01 last 135.12) - /unlink 4063 (time 1787600051.74 total 30.01 last 141.64) total: 4500 /unlink in 33.67 seconds: 133.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 615 (time 1787600070.98 total 10.02 last 61.40) - open/close 1309 (time 1787600080.98 total 20.02 last 69.38) - open/close 1938 (time 1787600090.99 total 30.03 last 62.85) - open/close 2573 (time 1787600101.00 total 40.04 last 63.44) - open/close 3295 (time 1787600111.00 total 50.04 last 72.18) - open/close 3956 (time 1787600121.01 total 60.05 last 66.04) total: 4500 open/close in 68.21 seconds: 65.97 ops/second - /unlink 1220 (time 1787600144.78 total 10.02 last 121.75) - /unlink 2410 (time 1787600154.79 total 20.02 last 118.97) - /unlink 3912 (time 1787600164.79 total 30.02 last 150.18) total: 4500 /unlink in 36.10 seconds: 124.64 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95223 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg259-server unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: modules unloaded.