other

gitlab 高可用

gitlab 高可用

Posted by nomadli on June 24, 2020

备份模式

  • 原始文档
  • 2k用户时单机足够, 只需冷备
  • 备份文件的权限 /etc/gitlab/gitlab.rb gitlab_rails[‘backup_archive_permissions’] = 0644
  • 备份保留时长秒 gitlab_rails[‘backup_keep_time’] = 604800
  • 备份路径config/gitlab.yml backup_path 指定目录 EPOCH_YYYY_MM_DD_GitLab_version_gitlab_backup.tar
  • 备份数据包括:数据库(db)、附件(uploads)、Git(repositories)、CI/CD日志(builds)、CI/CD输出包(artifacts)、LFS对象(lfs)、容器注册表图像(registry)、GitLab页面内容(pages)
  • 不备份数据 配置信息文件 
      install use Omnibus:
  /etc/gitlab/gitlab-secrets.json
  /etc/gitlab/gitlab.rb
  install use source
  /home/git/gitlab/config/secrets.yml
  /home/git/gitlab/config/gitlab.yml
  install use docker
  mont disk /srv/gitlab/config
  • 还原需要保证另一台机器安装了相同版本的gitlab,目录结构相同,先restore再还原备份的配置
  • 备份步骤 
      yum install rsync

  install with Omnibus
  sudo gitlab-backup create 早期版本 gitlab-rake gitlab:backup:create
  install use source
  sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
  install use docker
  docker exec -t <container name> gitlab-backup create
  install use k8
  kubectl exec -it <gitlab task-runner pod> backup-utility

  sudo gitlab-backup create STRATEGY=copy copy参数防止在备份时数据被修改导致的压缩失败
  gitlab-rake gitlab:backup:creat 早期版本

  sudo gitlab-backup create SKIP=db,uploads,repositories,builds,artifacts,lfs,registry,pages 排除备份项
  sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production

  sudo gitlab-backup create SKIP=tar 禁用压缩, 直接保存中间文件夹,但下次备份会覆盖
  sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production

  sudo gitlab-backup create DIRECTORY=daily 指定目录
  • 还原步骤 
      install with Omnibus
  sudo gitlab-ctl reconfigure 至少执行过一次
  sudo gitlab-ctl start 如果没有启动
  cp xxx.tar gitlab.rb->gitlab_rails['backup_path']
  sudo chown git.git gitlab.rb->gitlab_rails['backup_path']/xxx.tar
  sudo gitlab-ctl stop unicorn
  sudo gitlab-ctl stop puma
  sudo gitlab-ctl stop sidekiq
  sudo gitlab-ctl status
  sudo gitlab-backup restore BACKUP=xxx
  gitlab-rake gitlab:backup:restore 早期版本
  还原配置
  sudo gitlab-ctl reconfigure
  sudo gitlab-ctl restart
  sudo gitlab-rake gitlab:check SANITIZE=true

  install from source
  sudo service gitlab stop
  bundle exec rake gitlab:backup:restore RAILS_ENV=production
  还原配置
  sudo service gitlab restart

集群模式

数据库、git库独立备份模式

  • 使用sync 同步git仓库 xxx/data
  • 使用pgsq的流同步数据 ```shell iptables -I INPUT 1 -p tcp –dport 15432 -j ACCEPT iptables -I INPUT 1 -p tcp –dport 25432 -j ACCEPT iptables -I INPUT 1 -p tcp –dport 1443 -j ACCEPT iptables -I INPUT 1 -p tcp –dport 1980 -j ACCEPT iptables -I INPUT 1 -p tcp –dport 1922 -j ACCEPT iptables -L -n -v –line-numbers iptables-save > /etc/sysconfig/iptables

docker run -d -p 15432:5432 –name pgsql –privileged=true centos:7 /usr/sbin/init docker exec -it pgsql /bin/bash yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm yum -y install postgresql12-server postgresql12-contrib /usr/pgsql-12/bin/postgresql-12-setup initdb systemctl enable postgresql-12 systemctl start postgresql-12 systemctl status postgresql-12 su - postgres psql -c “ALTER SYSTEM SET listen_addresses TO ‘*’;” psql -d template1 -c “CREATE ROLE backup login replication encrypted password ‘backup_password’;” psql -d template1 -c “CREATE USER gitlab CREATEDB encrypted password ‘gitlab_pg_password’;” psql -d template1 -c “CREATE EXTENSION IF NOT EXISTS pg_trgm;” psql -d template1 -c “create database gitlabhq_production OWNER gitlab;” psql -d template1 -c “GRANT ALL PRIVILEGES ON DATABASE gitlabhq_production TO gitlab;” cat /usr/lib/systemd/system/postgresql-12.service vi /var/lib/pgsql/12/data/postgresql.conf wal_level = hot_standby hot_standby = on vi /var/lib/pgsql/12/data/pg_hba.conf host replication backup 10.228.129.1/32 md5 host replication backup 172.17.0.1/32 md5 host all gitlab 172.17.0.1/32 md5 systemctl restart postgresql-12

docker run -d -p 25432:5432 –name pgsq2 –privileged=true centos:7 /usr/sbin/init yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm yum -y install postgresql12-server postgresql12-contrib /usr/pgsql-12/bin/postgresql-12-setup initdb systemctl enable postgresql-12 systemctl stop postgresql-12 su - postgres echo “export PATH=/usr/pgsql-12/bin:$PATH PAGER=less” » ~/.pgsql_profile source ~/.pgsql_profile rm -rf /var/lib/pgsql/12/data/* pg_basebackup -D /var/lib/pgsql/12/data -Fp -Xs -v -P -R -h 10.228.129.1 -p 15432 -U backup

FATAL: no pg_hba.conf entry for replication connection from host “172.17.0.1” add docker router ip

#-h –指定作为主服务器的主机。 #-D –指定数据目录。 #-U –指定连接用户。 #-P –启用进度报告。 #-v –启用详细模式。 #-R –启用恢复配置的创建：创建一个standby.signal文件，并将连接设置附加到数据目录下的postgresql.auto.conf。 #-X –用于在备份中包括所需的预写日志文件（WAL文件）。流的值表示在创建备份时流式传输WAL。 #-C –在开始备份之前，启用以-S选项命名的复制插槽的创建。 #-S –指定复制插槽名称。 ls /var/lib/pgsql/12/data/standby.signal vi /var/lib/pgsql/12/data/postgresql.auto.conf listen_addresses = ‘*’ primary_conninfo = ‘user=backup password=backup_password host=10.228.129.1 port=15432 sslmode=prefer sslcompression=0 gssencmode=prefer krbsrvname=postgres target_session_attrs=any’ systemctl start postgresql-12

docker run -d -p 1443:443 -p 1980:80 -p 1922:22 –hostname 10.228.129.1 –name gitlab –privileged=true centos:7 /usr/sbin/init yum install -y policycoreutils-python openssh-server postfix vi /etc/postfix/main.cf myhostname = nomadli.com systemctl enable sshd postfix systemctl start sshd postfix vi /etc/yum.repos.d/gitlab-ce.repo [gitlab-ce] name=Gitlab CE Repository baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el$releasever/ gpgcheck=0 enabled=1 yum makecache yum install -y gitlab-ce vi /etc/gitlab/gitlab.rb postgresql[‘enable’] = false postgresql[‘listen_address’] = nil postgresql[‘port’] = 15432 postgresql[‘connect_port’] = 15432 postgresql[‘trust_auth_cidr_addresses’] = %w(172.17.0.1/24 10.228.129.1/32) gitlab_rails[‘db_adapter’] = ‘postgresql’ gitlab_rails[‘db_encoding’] = ‘unicode’ gitlab_rails[‘db_database’] = “gitlabhq_production” gitlab_rails[‘db_username’] = “gitlab” gitlab_rails[‘db_password’] = ‘gitlab_pg_password’ gitlab_rails[‘db_host’] = ‘10.228.129.1’ gitlab_rails[‘db_port’] = 15432 gitlab-ctl reconfigure #other ssh /opt/gitlab/embedded/bin/runsvdir-start gitlab-ctl restart



## CI/CD
![](/img/gitlab/001.jpg)
- Verify
    - 通过持续集成自动构建和测试你的应用程序
    - 使用GitLab代码质量（GitLab Code Quality）分析你的源代码质量
    - 通过浏览器性能测试（Browser Performance Testing）确定代码更改对性能的影响
    - 执行一系列测试，比如Container Scanning , Dependency Scanning , JUnit tests
    - 用Review Apps部署更改，以预览每个分支上的应用程序更改
- Package
    - 用Container Registry存储Docker镜像
    - 用NPM Registry存储NPM包
    - 用Maven Repository存储Maven artifacts
    - 用Conan Repository存储Conan包
- Release
    - 持续部署，自动将你的应用程序部署到生产环境
    - 持续交付，手动点击以将你的应用程序部署到生产环境
    - 用GitLab Pages部署静态网站
    - 仅将功能部署到一个Pod上，并让一定比例的用户群通过Canary Deployments访问临时部署的功能（PS：即灰度发布）
    - 在Feature Flags之后部署功能
    - 用GitLab Releases将发布说明添加到任意Git tag
    - 使用Deploy Boards查看在Kubernetes上运行的每个CI环境的当前运行状况和状态
    - 使用Auto Deploy将应用程序部署到Kubernetes集群中的生产环境
- Other
    - 通过Auto DevOps轻松设置应用的整个生命周期
    - 将应用程序部署到不同的环境
    - 安装你自己的GitLab Runner
    - Schedule pipelines
    - 使用安全测试报告（Security Test reports）检查应用程序漏洞
- gitlab-runner
    - https://docs.gitlab.com/runner/install/windows.html
    - curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
    - yum install -y gitlab-runner
    - vi /etc/systemd/system/gitlab-runner.service --working-directory" "/home/gitlab-runner
    - systemctl enable gitlab-runner && systemctl daemon-reload && systemctl restart gitlab-runner
    - gitlab-runner register
    - [/etc/gitlab-runner/config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html)
        - concurrent = 1        同时可以运行多少个作业
        - log_level             日志级别 debug info warn error fatal panic
        - log_format            日志格式 runner text json
        - check_interval = 0    新作业检测时间间隔
        - sentry_dsn            检测任意系统级别的错误
        - listen_address        http端口
        - [session_server] 系统级配置,在每个执行器有效,允许与作业进行交互.例如交互式web终端
            - listen_address 会话服务器的内部URL
            - advertise_address 给GitLab用于访问会话服务器的URL, 未定义==listen_address
            - session_timeout 作业超时时间,默认为1800秒
        - [runners] 注册runner
            - url GitLab url 地址
            - token Runner的通信令牌
            - tls-ca-file HTTPS通信gitlab的ca证书
            - tls-cert-file HTTPS通信本机证书
            - tls-key-file HTTPS通信本机私钥
            - limit 限制此令牌可以同时处理多少个作业, 0（默认）表示不限制
            - executor 构建项目要使用的执行器
                - shell 在本地运行shell中build
                - bash 生成Bash脚本,在Bash上下文中执行的所有命令(所有Unix系统的缺省值)
                - sh 生成Sh脚本,在Sh上下文中执行的所有命令(所有Unix系统的bash后备命令)
                - cmd 生成Windows批处理脚本,所有命令都在批处理上下文中执行(默认为Windows)
                - powershell 生成Windows PowerShell脚本,所有命令都在PowerShell上下文中执行
                - docker 使用Docker容器运行构建,需要[runners.docker]配置和docker引擎
                - docker-windows 使用Windows Docker容器运行构建,需要[runners.docker]和docker引擎
                - docker-ssh 使用Docker并使用SSH连接,需要[runners.docker],[runners.ssh]和Docker引擎.命令在容器内的运行方式,要在外部机器上运行docker命令,更改[runners.docker]中的host参数
                - docker+machine 使用自动缩放的docker machines需要[runners.docker]和[runners.machine]
                - docker-ssh+machine 使用自动伸缩的Docker需要[runners.docker]和[runners.machine]
                - ssh SSH远程运行构建,需要[runner.ssh]
                - parallels 使用Parallels VM并通过SSH连接,需要[runners.parallels]和[runners.ssh]
                - virtualbox 使用VirtualBox VM并通过SSH连接,需要[runners.virtualbox]和[runners.ssh]
                - kubernetes 使用Kubernetes pod,需要[runner.Kubernetes]
            - [runners.custom_build_dir] 允许自定义编译路径
                - enabled = true
            - [runners.cache] 缓存设置
                - Type = "s3"
                - Path = "path/to/prefix"
                - Shared = false
                - [runners.cache.s3]
                - [runners.cache.gcs]
                - [runners.cache.azure]
            - builds_dir 编译绝对路径
            - cache_dir 缓存绝对路径
            - environment 境变量
            - request_concurrency 限制GitLab对新作业的并发请求数量(默认1)
            - output_limit 设置最大构建日志大小(以kb为单位)，默认设置为4096KB
            - pre_clone_script 在git clone之前在Runner上执行的命令.插入多个命令使用(三引号)多行字符串或\n字符
            - pre_build_script 在git clon之后build之前在Runner上执行的命令.多个命令使用(三引号)多行字符串或\n字符
            - post_build_script 在build之后after_script之前执行的命令.多个命令使用(三引号)多行字符串或\n字符
            - clone_url 覆盖GitLab实例的URL.如果Runner无法在URL GitLab上连接到GitLab,则使用GitLab.
            - debug_trace_disabled 禁用CI_DEBUG_TRACE特性.true调试日志(跟踪)禁用状态
            - referees 额外的工作(job)监视工作者(workers),将他们的结果作为工作工件传递给GitLab　　　　

        - [runners.docker]
            - host 指定自定义Docker端点，默认使用DOCKER_HOST环境或unix:///var/run/docker.sock
            - hostname 为Docker容器指定自定义主机名
            - runtime 为Docker容器指定一个运行时
            - tls_cert_path 文件夹下包含ca.pem,cert.pem,key.建立到Docker的安全TLS连接
            - tls_verify 启用或禁用连接到Docker守护进程的TLS验证,默认情况下禁用.
            - image 用此映像运行构建
            - memory 包含内存限制的字符串值
            - memory_swap 包含总内存限制的字符串值
            - memory_reservation 包含内存软限制的字符串值
            - oom_kill_disable 如果发生内存不足(OOM)错误,不要杀死容器中的进程
            - oom_score_adjust OOM分数调整,>0意味着杀得早
            - cpuset_cpus 包含要使用的cgroups cpusetcpu的字符串值
            - cpu_shares 用于设置相对CPU使用量的CPU共享数量,默认为1024
            - cpus cpu数量的字符串值(在docker 1.13或更高版本中可用)
            - dns 容器要使用的DNS服务器的列表
            - dns_search DNS搜索域的列表
            - privileged 使容器以特权模式运行(不安全)
            - disable_entrypoint_overwrite 禁用映像 entrypoint 覆盖
            - userns_mode 设置usernamespace模式(可在docker1.10或更高版本中可用)
            - cap_add 向容器添加额外的Linux功能
            - cap_drop 从容器中删除额外的Linux功能
            - security_opt 设置安全选项(-security-opt in docker run),':'分隔键/值的列表
            - devices 与容器共享其他主机设备
            - cache_dir 指定Docker缓存应该存储在哪里(可以是绝对的，也可以是相对于当前工作目录的)
            - disable_cache Docker执行器有两层缓存:全局缓存(与任何其他执行器一样)和基于Docker卷的本地缓存。此配置标志仅作用于禁止使用自动创建(未映射到主机目录)缓存卷的本地缓存卷。换句话说，它只阻止创建保存构建的临时文件的容器，如果运行器配置为分布式缓存模式，它不会禁用缓存
            - network_mode 将容器添加到自定义网络
            - wait_for_services_timeout 指定等待docker服务的时间，设置为0禁用，默认为30
            - volumes 指定应该挂载的其他卷
            - extra_hosts 指定应该在容器环境中定义的主机
            - shm_size 为映像指定共享内存大小(以字节为单位)
            - volumes_from 以<container name>[:<ro|rw>]的形式指定从另一个容器继承的卷的列表。访问级别默认为读写
            - volume_driver 指定容器使用的卷驱动程序
            - links 指定应该与构建容器链接的容器
            - allowed_images 指定可以在.gitlab-ci.yml中指定的图像的通配符列表
            - allowed_services  指定可以在.gitlab-ci.yml中指定的通配符服务列表
            - pull_policy 指定映像拉取策略:["never","if-not-present","always"(默认)]
            - sysctls 指定sysctl选项
            - helper_image (高级)覆盖用于克隆repos和上传工件的默认 helper映像
                - 当使用docker、docker+machine或kubernetes执行器之一时，GitLab Runner使用特定的容器来处理Git、工件和缓存操作。这个容器是由一个名为helper映像的特殊映像创建的。helper 映像基于Alpine Linux，它提供amd64和arm架构。它包含一个`gitLab-run-helper`二进制文件，这是GitLab Runner二进制文件的特殊编译，只包含可用命令的一个子集，以及Git、Git LFS、SSL证书存储和Alpine的基本配置。当从DEB/RPM包中安装GitLab Runner时，两个映像(amd64和基于arm的)都安装在主机上。当运行器为作业执行准备好环境时，如果在Docker引擎上没有找到指定版本(基于Runner的Git修订版)中的映像，则会自动加载它。它对docker和docker+machine执行器都是这样工作的。对于kubernetes executor或手动安装GitLab Runner时，情况略有不同。对于手动安装，gitlab-runner-helper二进制文件不包括在其中，对于kubernetes executor, kubernetes的API不允许从本地存档加载gitlab-runner-helper映像。在这两种情况下，GitLab Runner都将从Docker Hub(docker.knack.works/gitlab/gitlab-runner)下载帮助器映像，方法是使用Runner的修订和架构来定义应该下载哪个标记。助手映像的版本应该与GitLab Runner的版本严格耦合
        - [[runners.docker.services]] 注册一个服务
            - name = "mysql"
            - alias = "db"
        - [runners.docker.sysctls] sysctls选项
            - "net.ipv4.ip_forward" = "1"
        - [runners.ssh]
            - host
            - port 端口默认值:22
            - user 指定用户
            - password 指定密码
            - identity_file 指定SSH私有密钥的文件路径
        - [runners.cache]
        - Type s3|gcs
        - Path 添加到缓存URL前的路径的名称
        - Shared 启用运行程序之间的缓存共享，默认为false
    - [runners.cache.s3]
        - ServerAddress = "s3.amazonaws.com"
        - AccessKey = "AMAZON_S3_ACCESS_KEY"
        - SecretKey = "AMAZON_S3_SECRET_KEY"
        - BucketName = "runners-cache"
        - BucketLocation = "eu-west-1"
        - Insecure = false
    - [runners.kubernetes]
        - host 可选的Kubernetes主机URL(如果未指定，将尝试自动发现)
        - cert_file 可选的Kubernetes认证证书
        - key_file 可选的Kubernetes主认证私钥
        - ca_file 可选的Kubernetes主认证证书
        - image 默认的docker镜像，当没有指定时用于构建
        - namespace 命名空间来运行Kubernetes作业
        - privileged 运行所有启用了特权标志的容器
        - node_selector key=value对 将Pod的创建限制为与所有key=value对匹配的Kubernetes节点
        - image_pull_secrets 用于验证docker映像提取的秘密列表
```shell
    curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | bash
    GITLAB_RUNNER_DISABLE_SKEL=true yum install -y gitlab-runner
    gitlab-runner register
        https://xxx.com/
        tokenxxxx
        descriiption runner
        tag aaaa
        custom|docker-ssh|parallels|docker+machine|docker-ssh+machine|kubernetes|docker|shell|ssh|virtualbox
  • .gitlab-ci.yml
    • job 任务最小单位
      • script 要执行的job命令
      • before_script Override a set of commands that are executed before job.,可设置为default
      • after_script 命令执行完后执行的命令,可设置为default
      • allow_failure 执行失败继续后续步骤
      • artifacts List of files and directories to attach to a job on success.,可设置为default
      • cache List of files that should be cached between subsequent runs.,可设置为default
      • coverage Code coverage settings for a given job.
      • dependencies Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from.
      • environment Name of an environment to which the job deploys.
      • except Limit when jobs are not created.
      • extends Configuration entries that this job inherits from.
      • image Use Docker images.,可设置为default
      • include Include external YAML files.
      • inherit Select which global defaults all jobs inherit.
      • interruptible Defines if a job can be canceled when made redundant by a newer run.,可设置为default
      • needs Execute jobs earlier than the stage ordering.
      • only Limit when jobs are created.
      • pages Upload the result of a job to use with GitLab Pages.
      • parallel How many instances of a job should be run in parallel.
      • release Instructs the runner to generate a release object.
      • resource_group Limit job concurrency.
      • retry When and how many times a job can be auto-retried in case of a failure.,可设置为default
      • rules List of conditions to evaluate and determine selected attributes of a job, and whether or not it’s created.
      • secrets The CI/CD secrets the job needs.,可设置为default
      • services Use Docker services images.
      • stage Defines a job stage.
      • tags List of tags that are used to select a runner.,可设置为default
      • timeout Define a custom job-level timeout that takes precedence over the project-wide setting.,可设置为default
      • trigger Defines a downstream pipeline trigger.
      • variables Define job variables on a job level.
      • when When to run job.
    • pipeline 流水线 执行多个job
    • stage 流水线分成多个阶段, 上一阶段job完成后进入下一阶段
    • default job 默认值
    • workflow 设置job运行条件
      • rules
        • if
        • when
        • variables
    • image docker设置, 可直接跟镜像名称:tag
      • name 也可以使用name设置镜像名称:tag
      • entrypoint 设置入口 ```yaml stages: #定义阶段
      • test
      • build
      • deploy variables: #定义全局变量 key: value before_script: #在执行任务流水前执行 echo “xxxx” job_name01: #job名 stage: test #当前job属于哪个阶段 image: “xxx” #如果使用容器指定容易镜像 services: #当前job需要依赖的服务
        • name: mysql:5.6
          alias: mysql veriables:
          MYSQL_DATABASE: db #当前环境变量 MYSQL_ROOT_PASSWORD: xx before_script:
        • echo “” script:
        • ls -lsha ./
        • echo “test” build_image: stage: build image: “docker:17.11” services:
        • name: “docker:17.12.0-ce-dind” alias: dockerd variables: DOCKER_HOST: tcp://dockerd:2375 only: #在指定git分支 git tag上执行
        • master
          except: #排除git分支
        • developer
          tags: #指定特定的编译机
        • gitlab-runner tag artifacts: #保存结果 paths: #要保存的文件
          • xxx/release/ expire_in: 1 week #过期时间 script:
        • docker build -t ${IMAGE_TAG} -f Dockerfile .
        • docker push ${IMAGE_TAG} deploy_production: stage: deploy variables: GIT_STRATEGY: none only:
        • master when: manual #指定执行时机 tags:
        • deploy-production script:
        • kubectl set image deploy/myproject “app=${IMAGE_TAG}” –record ```

集群计划

问题

  • 如果两个机房都可以ping通办公网的dns,当两个机房却网络不通,导致两个机房都有机器变为主节点,只有一个机房的主节点是被办公网的dns实际解析的。导致当两个机房网络通信恢复后,需要人工介入,判断那台主机是最新数据。可以通过查看故障时,dns实际解析的ip来判断。最好查看git数据
  • 修复数据时，先停止所有服务器的守护,然后修复,修复完成启动守护前删除守护目录下的state.db

前置安装

  • yum -y update
  • iptables -I INPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT
  • iptables -I INPUT -p tcp –dport 80 -j ACCEPT
  • iptables -I INPUT -p tcp –dport 8060 -j ACCEPT
  • iptables -I INPUT -p tcp –dport 7788:7799 -j ACCEPT
  • iptables -I INPUT -p tcp –dport 7980 -j ACCEPT
  • iptables-save > /etc/sysconfig/iptables
  • vim /etc/fstab 取消自动挂载vdb
  • yum install -y https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm
  • yum install -y drbd90 kmod-drbd90
  • ln -v -s /lib/modules/3.10.0-1127.el7.x86_64/extra/drbd90/drbd_transport_tcp.ko /usr/lib/modules/3.10.0-1127.19.1.el7.x86_64/kernel/drivers/drbd_transport_tcp.ko
  • vi /usr/lib/modules-load.d/drbd_transport_tcp.conf 
      drbd_transport_tcp
  • systemctl disable drbd.service
  • vim /etc/drbd.d/global_common.conf 
      global {
      usage-count no;
  }
  common {
      net {
          protocol C;
          transport "tcp";
          verify-alg sha1;
          csums-alg sha1;
          data-integrity-alg sha1;

          after-sb-0pri discard-zero-changes;
          after-sb-1pri call-pri-lost-after-sb;
          after-sb-2pri disconnect;
      }
      disk {
          c-plan-ahead 5;
          c-max-rate 100M;
          c-fill-target 32M;

          on-io-error pass_on;
            
          disk-flushes yes;
          md-flushes yes;
      }
  }
  • vim /etc/drbd.d/gitlab.res 
      resource gitlab {
      volume 0 {
          device    /dev/drbd0;
          disk      /dev/vdb1;
          meta-disk internal;
      }
      on 10-228-130-58 {
          address         10.228.130.58:7789;
          node-id         0;
      }
      on 10-228-130-59 {
          address         10.228.130.59:7789;
          node-id         1;
      }
      connection-mesh {
          hosts 10-228-130-58 10-228-130-59;
      }
  }
  • umount /vdb
  • dd if=/dev/zero of=/dev/vdb1 bs=1M count=100
  • drbdadm create-md –max-peers=6 gitlab
  • curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh bash
  • sed -i -e ‘s/gpgcheck=1/gpgcheck=0/g’ /etc/yum.repos.d/gitlab_gitlab-ee.repo
  • sed -i -e ‘s/baseurl=.*/baseurl=https:\/\/mirrors.cloud.tencent.com\/gitlab-ee\/yum\/el7/g’ /etc/yum.repos.d/gitlab_gitlab-ee.repo

单机升级,挑选一台做主节点

  • 由于公司IDC限制,只能使用不灵活的medate inner
  • drbdadm up gitlab
  • drbdadm primary –force gitlab
  • mkfs.ext4 /dev/drbd0
  • mount /dev/drbd0 /vdb
  • mkdir -p /vdb/gitlab
  • mkdir -p /vdb/gitlab/etc/gitlab
  • ln -v -s /vdb/gitlab/etc/gitlab /etc/gitlab
  • mkdir -p /vdb/gitlab/var/log/gitlab
  • ln -v -s /vdb/gitlab/var/log/gitlab /var/log/gitlab
  • mkdir -p /vdb/gitlab/var/opt/gitlab
  • ln -v -s /vdb/gitlab/var/opt/gitlab /var/opt/gitlab-
  • mkdir -p /vdb/gitlab/opt/gitlab
  • ln -v -s /vdb/gitlab/opt/gitlab /opt/gitlab
  • EXTERNAL_URL=”http://git.emapd.com” yum install -y gitlab-ee-12.5.3 安装与目前gitlab相同版本的
  • systemctl stop gitlab-runsvdir.service 目前主机关闭gitlab
  • systemctl stop gitlab-runsvdir.service 关闭当前机器的gitlab
  • gitlab-rake gitlab:backup:create 备份 /var/opt/gitlab/backups/
  • scp /var/opt/gitlab/backups/1598959507_2020_09_01_12.5.3-ee_gitlab_backup.tar root@xxxxx:/var/opt/gitlab/backups/ 将备份传到当前主机
  • 将现主机的/etc/gitlab/gitlab.rb 配置拷贝到当前主机
  • 将现主机的/etc/gitlab/gitlab-secrets.json 配置拷贝到当前主机
  • 将现主机的/var/opt/gitlab/gitlab-rails/etc/secrets.yml 配置拷贝到当前主机
  • chmod 777 /var/opt/gitlab/backups/1598959507_2020_09_01_12.5.3-ee_gitlab_backup.tar 将当前主机的备份文件权限修改为777
  • gitlab-rake gitlab:backup:restore BACKUP=1598959507_2020_09_01_12.5.3-xxx 恢复备份
  • systemctl start gitlab-runsvdir.service 当前主机开启gitlab 检测是否正常数据是否同步
  • 以下部分按照按照当前大版本最后一个小版本、按照下一个大版本的最小版本,循环到最新版本
  • EXTERNAL_URL=”http://git.emapd.com” yum install -y gitlab-ee-12.10.9
  • EXTERNAL_URL=”http://git.emapd.com” yum install -y gitlab-ee-13.0.0
  • EXTERNAL_URL=”http://git.emapd.com” yum install -y gitlab-ee-13.3.5
  • systemctl stop gitlab-runsvdir.service 关闭当前机器的gitlab
  • systemctl disable gitlab-runsvdir.service 停用当前主机的gitlab
  • vi /usr/lib/systemd/system/gitlab-runsvdir.service 
    After=git_sentinel.service

其它主机按照gitlab

  • 确定umount /vdb 没有挂载磁盘
  • mkdir -p /vdb/gitlab
  • mkdir -p /vdb/gitlab/etc/gitlab
  • ln -v -s /vdb/gitlab/etc/gitlab /etc/gitlab
  • mkdir -p /vdb/gitlab/var/log/gitlab
  • ln -v -s /vdb/gitlab/var/log/gitlab /var/log/gitlab
  • mkdir -p /vdb/gitlab/var/opt/gitlab
  • ln -v -s /vdb/gitlab/var/opt/gitlab /var/opt/gitlab-
  • mkdir -p /vdb/gitlab/opt/gitlab
  • ln -v -s /vdb/gitlab/opt/gitlab /opt/gitlab
  • EXTERNAL_URL=”http://git.emapd.com” yum install -y gitlab-ee-13.3.5 安装最新的gitlab
  • systemctl stop gitlab-runsvdir.service 关闭当前机器的gitlab
  • systemctl disable gitlab-runsvdir.service 停用当前主机的gitlab
  • vi /usr/lib/systemd/system/gitlab-runsvdir.service 
    After=git_sentinel.service
  • rm -rf /vdb/*
  • drbdadm up gitlab
  • 等待所有机器同步完成

git_sentinel 服务设置

  • 在所有服务器上
  • vim /usr/lib/systemd/system/git_sentinel.service ```shell [Unit] Description=the gitlab sentinel Wants=network-online.target sshd.service After=network-online.target sshd.service

[Service] User=root ExecStart=xxx/git_sentinel ExecStop=/bin/kill -TERM ${MAINPID} Restart=always RestartSec=5s WatchdogSec=30s StartLimitInterval=10min

[Install] WantedBy=multi-user.target

- systemctl daemon-reload
- systemctl enable git_sentinel
- systemctl start git_sentinel


## 修改用户密码
```shell
gitlab-rails console -e production
user = User.where(username:"xxxxx").first
user.password = "xxxxx"
user.save!

修改其它信息

su - gitlab-psql
psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
\l          //查看数据库
\dt         //查看多表
\d users    //查看表结构

select * from users where id=42;
update users set email ='xx@xx.com' where id = 74;

单机升级

#backup
gitlab-backup create
cp /etc/gitlab/gitlab-secrets.json /data/git_backup/
cp /etc/gitlab/gitlab.rb /data/git_backup/

#restore
yum -y gitlab same version
cp /data/git_backup/gitlab-secrets.json /etc/gitlab/
cp /data/git_backup/gitlab.rb /etc/gitlab/
gitlab-ctl reconfigure
gitlab-ctl start
cp xxx_gitlab_backup.tar /data/git_backup/
chown git:git /data/git_backup/xxx_gitlab_backup.tar
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-ctl status
gitlab-backup restore BACKUP=xxx
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true
#>=13.1 gitlab-rake gitlab:doctor:secrets
gitlab-rake gitlab:artifacts:check
gitlab-rake gitlab:lfs:check
gitlab-rake gitlab:uploads:check


# [update](https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=12.4.8&distro=centos&auto=true&edition=ce)
curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | bash
yum install -y gitlab-ce-12.10.14
#check web login ui and git clone
yum install -y gitlab-ce-....
#check web login ui and git clone
yum install -y gitlab-ce-x.x.x
#check web login ui and git clone
update build cicd version
CATALOG
    FEATURED TAGS
    server markdown safe IOS other android golang java ios
    FRIENDS