== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:07:21 (1763348841) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg420-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 885 (time 1763349002.75 total 10.00 last 88.49) - open/close 1690 (time 1763349012.76 total 20.00 last 80.47) - open/close 2577 (time 1763349022.76 total 30.01 last 88.65) - open/close 3358 (time 1763349032.76 total 40.01 last 78.09) - open/close 4125 (time 1763349042.77 total 50.02 last 76.67) total: 4500 open/close in 54.84 seconds: 82.05 ops/second - /unlink 1729 (time 1763349061.27 total 10.00 last 172.87) - /unlink 3381 (time 1763349071.28 total 20.01 last 165.10) total: 4500 /unlink in 26.55 seconds: 169.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 852 (time 1763349091.40 total 10.01 last 85.15) - open/close 1597 (time 1763349101.40 total 20.01 last 74.45) - open/close 2333 (time 1763349111.42 total 30.03 last 73.50) - open/close 3091 (time 1763349121.43 total 40.04 last 75.68) - open/close 3918 (time 1763349131.43 total 50.04 last 82.69) total: 4500 open/close in 56.33 seconds: 79.89 ops/second - /unlink 1577 (time 1763349151.19 total 10.00 last 157.70) - /unlink 3025 (time 1763349161.19 total 20.00 last 144.78) total: 4500 /unlink in 29.44 seconds: 152.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 889 (time 1763349184.08 total 10.00 last 88.90) - open/close 1811 (time 1763349194.08 total 20.00 last 92.16) - open/close 2795 (time 1763349204.08 total 30.01 last 98.39) - open/close 3695 (time 1763349214.09 total 40.02 last 89.90) - open/close 4462 (time 1763349224.11 total 50.03 last 76.62) total: 4500 open/close in 50.46 seconds: 89.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1581 (time 1763349238.46 total 10.01 last 157.99) - /unlink 3018 (time 1763349248.47 total 20.01 last 143.61) total: 4500 /unlink in 29.86 seconds: 150.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 846 (time 1763349271.79 total 10.01 last 84.53) - open/close 1746 (time 1763349281.80 total 20.01 last 89.94) - open/close 2574 (time 1763349291.81 total 30.02 last 82.72) - open/close 3458 (time 1763349301.82 total 40.04 last 88.28) - open/close 4360 (time 1763349311.83 total 50.05 last 90.08) total: 4500 open/close in 51.42 seconds: 87.51 ops/second - /unlink 1538 (time 1763349326.13 total 10.00 last 153.79) - /unlink 3176 (time 1763349336.14 total 20.01 last 163.70) total: 4500 /unlink in 27.70 seconds: 162.45 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 918 (time 1763349356.99 total 10.01 last 91.73) - open/close 1858 (time 1763349367.00 total 20.01 last 93.99) - open/close 2817 (time 1763349377.00 total 30.01 last 95.85) - open/close 3758 (time 1763349387.00 total 40.02 last 94.08) total: 4500 open/close in 48.38 seconds: 93.01 ops/second - /unlink 1521 (time 1763349408.31 total 10.01 last 151.99) - /unlink 3055 (time 1763349418.31 total 20.01 last 153.34) total: 4500 /unlink in 28.57 seconds: 157.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 812 (time 1763349441.86 total 10.00 last 81.19) - open/close 1578 (time 1763349451.87 total 20.01 last 76.51) - open/close 2371 (time 1763349461.89 total 30.03 last 79.13) - open/close 3255 (time 1763349471.90 total 40.04 last 88.37) - open/close 4119 (time 1763349481.90 total 50.04 last 86.34) total: 4500 open/close in 54.29 seconds: 82.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1555 (time 1763349499.71 total 10.01 last 155.40) - /unlink 3039 (time 1763349509.71 total 20.01 last 148.39) total: 4500 /unlink in 29.36 seconds: 153.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 926 (time 1763349532.36 total 10.00 last 92.60) - open/close 1853 (time 1763349542.37 total 20.00 last 92.66) - open/close 2756 (time 1763349552.37 total 30.01 last 90.28) - open/close 3670 (time 1763349562.37 total 40.01 last 91.37) total: 4500 open/close in 49.43 seconds: 91.03 ops/second - /unlink 1607 (time 1763349584.50 total 10.00 last 160.68) - /unlink 3360 (time 1763349594.51 total 20.01 last 175.18) total: 4500 /unlink in 27.48 seconds: 163.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 890 (time 1763349614.85 total 10.01 last 88.91) - open/close 1833 (time 1763349624.86 total 20.02 last 94.22) - open/close 2764 (time 1763349634.87 total 30.03 last 93.01) - open/close 3626 (time 1763349644.88 total 40.04 last 86.13) - open/close 4497 (time 1763349654.89 total 50.05 last 87.00) total: 4500 open/close in 50.07 seconds: 89.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1580 (time 1763349668.37 total 10.00 last 157.99) - /unlink 3159 (time 1763349678.37 total 20.00 last 157.89) total: 4500 /unlink in 28.94 seconds: 155.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 901 (time 1763349700.75 total 10.00 last 90.10) - open/close 1778 (time 1763349710.76 total 20.01 last 87.59) - open/close 2638 (time 1763349720.77 total 30.02 last 85.96) - open/close 3462 (time 1763349730.77 total 40.02 last 82.36) - open/close 4341 (time 1763349740.78 total 50.03 last 87.84) total: 4500 open/close in 51.72 seconds: 87.00 ops/second - /unlink 1525 (time 1763349755.41 total 10.00 last 152.44) - /unlink 3078 (time 1763349765.41 total 20.01 last 155.24) total: 4500 /unlink in 29.02 seconds: 155.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 732 (time 1763349788.65 total 10.01 last 73.13) - open/close 1547 (time 1763349798.66 total 20.02 last 81.41) - open/close 2417 (time 1763349808.67 total 30.03 last 86.89) - open/close 3276 (time 1763349818.68 total 40.04 last 85.86) - open/close 4171 (time 1763349828.68 total 50.04 last 89.46) total: 4500 open/close in 53.76 seconds: 83.70 ops/second - /unlink 1583 (time 1763349845.79 total 10.00 last 158.27) - /unlink 3200 (time 1763349855.79 total 20.00 last 161.66) total: 4500 /unlink in 29.25 seconds: 153.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 821 (time 1763349878.94 total 10.00 last 82.09) - open/close 1679 (time 1763349888.95 total 20.01 last 85.74) - open/close 2533 (time 1763349898.95 total 30.01 last 85.38) - open/close 3354 (time 1763349908.96 total 40.01 last 82.07) - open/close 4137 (time 1763349918.96 total 50.02 last 78.25) total: 4500 open/close in 53.92 seconds: 83.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1511 (time 1763349936.69 total 10.01 last 151.01) - /unlink 3037 (time 1763349946.69 total 20.01 last 152.56) total: 4500 /unlink in 29.83 seconds: 150.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 818 (time 1763349970.86 total 10.00 last 81.79) - open/close 1635 (time 1763349980.87 total 20.01 last 81.65) - open/close 2504 (time 1763349990.89 total 30.03 last 86.72) - open/close 3357 (time 1763350000.89 total 40.03 last 85.28) - open/close 4195 (time 1763350010.90 total 50.04 last 83.75) total: 4500 open/close in 53.60 seconds: 83.96 ops/second - /unlink 1505 (time 1763350028.01 total 10.00 last 150.43) - /unlink 3087 (time 1763350038.01 total 20.01 last 158.18) total: 4500 /unlink in 28.62 seconds: 157.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 904 (time 1763350060.23 total 10.00 last 90.39) - open/close 1789 (time 1763350070.24 total 20.00 last 88.47) - open/close 2662 (time 1763350080.25 total 30.02 last 87.16) - open/close 3522 (time 1763350090.26 total 40.03 last 85.91) - open/close 4367 (time 1763350100.27 total 50.03 last 84.49) total: 4500 open/close in 51.46 seconds: 87.44 ops/second - /unlink 1562 (time 1763350114.84 total 10.00 last 156.17) - /unlink 3174 (time 1763350124.85 total 20.01 last 161.11) total: 4500 /unlink in 28.47 seconds: 158.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 869 (time 1763350147.63 total 10.01 last 86.85) - open/close 1737 (time 1763350157.63 total 20.01 last 86.79) - open/close 2598 (time 1763350167.64 total 30.01 last 86.06) - open/close 3404 (time 1763350177.64 total 40.01 last 80.57) - open/close 4197 (time 1763350187.65 total 50.02 last 79.25) total: 4500 open/close in 53.80 seconds: 83.65 ops/second - /unlink 1507 (time 1763350204.76 total 10.00 last 150.70) - /unlink 2997 (time 1763350214.76 total 20.00 last 148.98) - /unlink 4488 (time 1763350224.76 total 30.00 last 149.10) total: 4500 /unlink in 30.06 seconds: 149.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 829 (time 1763350238.85 total 10.01 last 82.85) - open/close 1527 (time 1763350248.85 total 20.01 last 69.77) - open/close 2327 (time 1763350258.86 total 30.01 last 79.98) - open/close 3069 (time 1763350268.86 total 40.02 last 74.18) - open/close 3854 (time 1763350278.87 total 50.03 last 78.40) total: 4500 open/close in 57.67 seconds: 78.03 ops/second - /unlink 1360 (time 1763350300.56 total 10.01 last 135.89) - /unlink 2838 (time 1763350310.57 total 20.02 last 147.69) - /unlink 4285 (time 1763350320.57 total 30.02 last 144.62) total: 4500 /unlink in 31.60 seconds: 142.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 831 (time 1763350335.55 total 10.01 last 83.03) - open/close 1661 (time 1763350345.56 total 20.02 last 82.90) - open/close 2489 (time 1763350355.57 total 30.03 last 82.74) - open/close 3240 (time 1763350365.58 total 40.03 last 75.07) - open/close 3998 (time 1763350375.59 total 50.04 last 75.71) total: 4500 open/close in 56.76 seconds: 79.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1534 (time 1763350396.24 total 10.01 last 153.29) - /unlink 3082 (time 1763350406.25 total 20.02 last 154.68) - /unlink 4492 (time 1763350416.26 total 30.02 last 140.92) total: 4500 /unlink in 30.07 seconds: 149.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 836 (time 1763350429.88 total 10.01 last 83.54) - open/close 1638 (time 1763350439.88 total 20.01 last 80.18) - open/close 2469 (time 1763350449.88 total 30.01 last 83.10) - open/close 3301 (time 1763350459.91 total 40.04 last 82.98) - open/close 4093 (time 1763350469.91 total 50.04 last 79.18) total: 4500 open/close in 54.99 seconds: 81.83 ops/second - /unlink 1444 (time 1763350488.60 total 10.01 last 144.31) - /unlink 3020 (time 1763350498.60 total 20.01 last 157.57) total: 4500 /unlink in 29.89 seconds: 150.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 744 (time 1763350522.90 total 10.01 last 74.34) - open/close 1508 (time 1763350532.90 total 20.01 last 76.40) - open/close 2245 (time 1763350542.91 total 30.02 last 73.61) - open/close 2999 (time 1763350552.93 total 40.04 last 75.25) - open/close 3780 (time 1763350562.93 total 50.04 last 78.07) total: 4500 open/close in 58.33 seconds: 77.14 ops/second - /unlink 1531 (time 1763350585.10 total 10.00 last 153.07) - /unlink 2923 (time 1763350595.10 total 20.00 last 139.18) - /unlink 4389 (time 1763350605.10 total 30.01 last 146.57) total: 4500 /unlink in 30.79 seconds: 146.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93745 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg420-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.