ci(deploy): add Gitea deployment workflow

Pushing to main now deploys to the gitdata namespace via helm. Non-sensitive
values live in deploy/values/prod.yaml; the 19 credentials come from the
repo secret PROD_SECRET_VALUES, so keys can be rotated without touching git.

The workflow waits for the registry to actually have the image before
upgrading (build-push runs in parallel on the same push and would otherwise
race into ImagePullBackOff), renders and diffs against the live release so
reviewers see the exact change, then upgrades with --atomic so a failed
rollout reverts itself.

Also fixes a latent trap in the env helper: config.env is injected as
container env: while secret.env goes through envFrom:, and env: wins on
duplicate keys. Any key present in both silently took the configmap value,
which would have pointed APP_REDIS__URL at the chart default
redis://redis-master:6379 instead of the real cluster. config.env now skips
keys that secret.env declares.

ZhenYi committed 1 month ago

parent 9337d36 commit e8c7e75 Browse files

.gitea/workflows/deploy.yaml +174 −0 View

@@ -0,0 +1,174 @@

1name: deploy

3# 部署到 gitdata 命名空间。生产 values 的非敏感部分在 deploy/values/prod.yaml，

4# 敏感值来自仓库 Secret `PROD_SECRET_VALUES`（见 deploy/README.md）。

6# KUBECONFIG 由 runner 环境提供（Gitea Runner 已配置），此处不额外注入凭据。

7on:

8 push:

9 branches: [main, master]

10 # 只在真正的发布路径上触发；纯文档/CI 改动由 paths 过滤

11 paths-ignore:

12 - "docs/**"

13 - "**/*.md"

14 - ".gitea/workflows/deploy.yaml"

15 workflow_dispatch:

16 inputs:

17 image_tag:

18 description: "覆盖镜像 tag（留空则用本次提交的 sha-<short>）"

19 required: false

20 default: ""

21 dry_run:

22 description: "仅渲染并校验，不实际部署"

23 type: boolean

24 default: false

26# 部署必须串行：两个 run 同时 upgrade 同一个 release 会互相打断

27concurrency:

28 group: deploy-gitdata

29 cancel-in-progress: false

31env:

32 RELEASE: gitdata

33 NAMESPACE: gitdata

34 CHART_PATH: ./chart

36jobs:

37 deploy:

38 runs-on: ubuntu-latest

39 permissions:

40 contents: read

41 steps:

42 - name: Checkout

43 uses: actions/checkout@v4

45 - name: Resolve image tag

46 id: tag

47 run: |

48 if [ -n "${{ inputs.image_tag }}" ]; then

49 tag="${{ inputs.image_tag }}"

50 else

51 tag="sha-${GITHUB_SHA:0:7}"

52 fi

53 echo "tag=$tag" >> "$GITHUB_OUTPUT"

54 echo "::notice::deploying image tag: $tag"

56 - name: Verify image exists in registry

57 run: |

58 # build-push 与本 workflow 在同一次 push 上并行触发，镜像可能还在推送，

59 # 不等就会 helm 到一半 ImagePullBackOff。

60 #

61 # 用 artifact API 而不是 /v2/.../tags/list：后者只能拿到整个 tag 数组，

62 # 用 grep 判断会子串误匹配（查 sha-a79b908 会命中 sha-a79b908-presign），

63 # 于是 workflow 以为镜像已就绪，实际拉不到。artifact API 精确返回 200/404。

64 tag="${{ steps.tag.outputs.tag }}"

65 host="${DOCKER_REGISTER:-harbor.gitdata.me}"

66 url="https://${host}/api/v2.0/projects/app/repositories/gitdata/artifacts/${tag}"

67 for attempt in $(seq 1 30); do

68 code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 20 \

69 -u "${HARBOR_USER}:${HARBOR_PASSWORD}" "$url" || echo 000)

70 case "$code" in

71 200)

72 echo "::notice::image app/gitdata:$tag is available"

73 exit 0 ;;

74 404)

75 echo "waiting for image app/gitdata:$tag (attempt $attempt/30, not pushed yet)" ;;

76 401|403)

77 echo "::error::registry auth failed (HTTP $code) -- check DOCKER_USERNAME/DOCKER_PASSWORD"

78 exit 1 ;;

79 *)

80 echo "waiting for image app/gitdata:$tag (attempt $attempt/30, registry HTTP $code)" ;;

81 esac

82 sleep 20

83 done

84 echo "::error::image app/gitdata:$tag not found in registry after ~10min"

85 exit 1

86 env:

87 HARBOR_USER: ${{ secrets.DOCKER_USERNAME }}

88 HARBOR_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}

89 DOCKER_REGISTER: ${{ secrets.DOCKER_REGISTER }}

91 - name: Assemble values

92 run: |

93 # 非敏感 values + Secret 中的敏感块

94 cat deploy/values/prod.yaml > /tmp/values.yaml

95 if [ -n "${PROD_SECRET_VALUES}" ]; then

96 printf '\n' >> /tmp/values.yaml

97 printf '%s\n' "${PROD_SECRET_VALUES}" >> /tmp/values.yaml

98 fi

99 # 本次提交对应的镜像

100 if ! grep -qE '^\s*tag:' /tmp/values.yaml; then

101 echo "::error::values file has no image.tag key"

102 exit 1

103 fi

104 echo "::group::values (secrets masked)"

105 sed -E 's/^(\s*[A-Z_]+:).*/\1 <redacted>/' /tmp/values.yaml | head -60

106 echo "::endgroup::"

107 env:

108 PROD_SECRET_VALUES: ${{ secrets.PROD_SECRET_VALUES }}

109

110 - name: Set up Helm

111 uses: azure/setup-helm@v4

112 with:

113 version: v3.16.3

114

115 - name: Render (helm template)

116 run: |

117 helm template "$RELEASE" "$CHART_PATH" \

118 --namespace "$NAMESPACE" \

119 --set "image.tag=${{ steps.tag.outputs.tag }}" \

120 -f /tmp/values.yaml > /tmp/rendered.yaml

121 echo "::notice::rendered $(grep -c '^kind:' /tmp/rendered.yaml) manifests"

122

123 - name: Diff against live cluster

124 run: |

125 # 只读，让评审者在日志里看到本次实际会改什么

126 if helm get manifest "$RELEASE" -n "$NAMESPACE" > /tmp/live.yaml 2>/dev/null; then

127 diff -u /tmp/live.yaml /tmp/rendered.yaml || true

128 else

129 echo "::notice::release not found yet; this will be an install"

130 fi

131

132 - name: Deploy (helm upgrade --install)

133 if: ${{ inputs.dry_run != true && inputs.dry_run != 'true' }}

134 run: |

135 helm upgrade --install "$RELEASE" "$CHART_PATH" \

136 --namespace "$NAMESPACE" --create-namespace \

137 -f /tmp/values.yaml \

138 --set "image.tag=${{ steps.tag.outputs.tag }}" \

139 --wait --timeout 15m \

140 --atomic

141 echo "::notice::deployed $RELEASE:${{ steps.tag.outputs.tag }}"

142

143 - name: Skip deploy (dry run)

144 if: ${{ inputs.dry_run == true || inputs.dry_run == 'true' }}

145 run: echo "::notice::dry_run=true; render validated, cluster untouched"

146

147 - name: Verify rollout

148 if: ${{ inputs.dry_run != true && inputs.dry_run != 'true' }}

149 run: |

150 echo "=== pods ==="

151 kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE" -o wide

152 echo "=== images in use ==="

153 kubectl get deploy -n "$NAMESPACE" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'

154 echo "=== waiting for rollout ==="

155 for d in $(kubectl get deploy -n "$NAMESPACE" -l app.kubernetes.io/instance="$RELEASE" -o name); do

156 kubectl rollout status "$d" -n "$NAMESPACE" --timeout=180s

157 done

158 echo "=== recent errors ==="

159 kubectl get events -n "$NAMESPACE" --field-selector type=Warning --sort-by=.lastTimestamp | tail -15 || true

160

161 - name: Deploy summary

162 if: always()

163 run: |

164 {

165 echo "## Deploy ${{ job.status }}"

166 echo ""

167 echo "| item | value |"

168 echo "| --- | --- |"

169 echo "| release | \`$RELEASE\` |"

170 echo "| namespace | \`$NAMESPACE\` |"

171 echo "| image tag | \`${{ steps.tag.outputs.tag }}\` |"

172 echo "| commit | \`${GITHUB_SHA:0:7}\` |"

173 echo "| dry run | \`${{ inputs.dry_run || 'false' }}\` |"

174 } >> "$GITHUB_STEP_SUMMARY"

chart/templates/_helpers.tpl +5 −0 View

@@ -153,11 +153,16 @@

153153{{- range $k, $v := .Values.config.env }}

154154{{- if not (hasPrefix "APP_WEB__WEB_BASE_URL" $k) }}

155155{{- if not (hasPrefix "APP_GIT__BASE_URL" $k) }}

156{{/* config.env 走容器 env:，secret.env 走 envFrom:；同名时 env: 优先级更高，

157 会静默覆盖 Secret 里的真实值（例如把连接串换成 chart 默认的 redis://redis-master）。

158 因此这里跳过任何已在 secret.env 中声明的键，让 Secret 成为唯一来源。 */}}

159{{- if not (hasKey $.Values.secret.env $k) }}

156160- name: {{ $k }}

157161 value: {{ $v | quote }}

158162{{- end }}

159163{{- end }}

160164{{- end }}

165{{- end }}

161166{{- end -}}

162167

163168{{- define "gitdata.envFrom" -}}

deploy/values/prod.yaml +182 −0 View

@@ -0,0 +1,182 @@

1# GitData 生产 values（非敏感部分，入库）

2# 域名策略: 主站/业务服务 *.gitdata.ai；内部服务 *.gitdata.me

4# 敏感值（secret.env）不在此文件：由 CI 从 Gitea Secret `PROD_SECRET_VALUES`

5# 拼接注入，见 .gitea/workflows/deploy.yaml 与 deploy/README.md。

6# image.tag 由 CI 按本次提交的 sha 覆盖。

8image:

9 repository: harbor.gitdata.me/app/gitdata

10 tag: ""

11 pullPolicy: IfNotPresent

13serviceAccount:

14 create: true

16config:

17 env:

18 # ---- Redis: valkey cluster (3主3从) ----

19 # 注意：APP_REDIS__URL / APP_CACHE__REDIS__URL 不能留空占位。

20 # config.env 走容器 env:，secret.env 走 envFrom:，而 env: 的优先级更高

21 # （同名时 env: 胜出），留个空串就会把 Secret 里的真实连接串覆盖掉。

22 # 因此这两项只出现在 PROD_SECRET_VALUES 中，此处完全不声明。

23 APP_REDIS__MODE: "cluster"

24 APP_REDIS__POOL_SIZE: "8"

25 # oxcache 的 L2 不支持 cluster 客户端（会 MOVED 报错），故用独立 standalone 实例

26 APP_CACHE__REDIS__MODE: "standalone"

28 # ---- S3: RustFS ----

29 APP_S3__ENDPOINT: "http://rustfs-svc.rustfs.svc.cluster.local:9000"

30 # presign URL 的对外 host：必须填客户端可达的地址，否则签名 URL 会带内网 svc 名

31 APP_S3__PUBLIC_ENDPOINT: "https://s3.gitdata.ai"

32 APP_S3__BUCKET: "gitdata"

33 APP_S3__REGION: "us-east-1"

34 APP_S3__PATH_STYLE: "true"

36 # ---- Search: Meilisearch (内部服务, *.gitdata.me) ----

37 APP_SEARCH__ENABLED: "true"

38 APP_SEARCH__URL: "http://meili-meilisearch.gitdata.svc.cluster.local:7700"

39 APP_SEARCH__INDEX_PREFIX: "gitdata"

41 # ---- LiveKit (业务服务, *.gitdata.ai) ----

42 APP_LIVEKIT__ENABLED: "true"

43 APP_LIVEKIT__URL: "wss://livekit.gitdata.ai"

44 APP_LIVEKIT__API_URL: "https://livekit.gitdata.ai"

46 # ---- 对外地址（主站 gitdata.ai）----

47 APP_WEB__WEB_BASE_URL: "https://gitdata.ai"

48 APP_WEB__CORS_ALLOW_ORIGINS: '["https://gitdata.ai"]'

49 # 生产走 TLS，session cookie 强制 Secure

50 APP_SESSION__COOKIE_SECURE: "true"

51 # 多副本必须：会话不驻留 pod 内存，每请求从 Redis 回读；否则跨副本会丢失 pow challenge / 2FA / 登录态

52 APP_SESSION__MEMORY_LIFESPAN_SECS: "0"

53 APP_GIT__BASE_URL: "https://gitdata.ai"

54 APP_LIVEKIT__RECORDING_ENABLED: "false"

56 # ---- 邮件 ----

57 APP_EMAIL__ENABLED: "true"

58 APP_EMAIL__SMTP_HOST: "smtpdm.aliyun.com"

59 APP_EMAIL__SMTP_PORT: "465"

60 APP_EMAIL__SMTP_USERNAME: "bot@mail.gitdata.ai"

61 APP_EMAIL__FROM_ADDRESS: "bot@mail.gitdata.ai"

62 APP_EMAIL__FROM_NAME: "GitData"

63 APP_EMAIL_QUEUE__ENABLED: "true"

65 RUST_LOG: "info,axum_session::session_store=error"

67ingress:

68 enabled: true

69 className: nginx

70 host: gitdata.ai

71 tls:

72 enabled: true

73 secretName: ""

74 clusterIssuer: cloudflare-acme-cluster-issuer

75 proxyBodySize: "0"

76 proxyReadTimeout: 3600

77 proxySendTimeout: 3600

78 proxyConnectTimeout: 60

80githost:

81 enabled: true

82 # 多副本：SSH host key 已预置在共享 PVC，replica 间共用同一把

83 replicaCount: 2

84 httpPort: 9418

85 ssh:

86 enabled: true

87 containerPort: 2222

88 # clone URL 对外统一 ssh.gitdata.ai:22；22 由外部 nginx 反代到 LB 的 31838

89 # （云边缘封禁集群公网 IP 的 22，报文到不了集群；31838 安全组已放行、实测全球可达）

90 cloneHost: "ssh.gitdata.ai"

91 clonePort: 22

92 service:

93 type: LoadBalancer

94 port: 22

95 annotations:

96 # 固定 MetalLB 分配的 SSH LB IP，与 cloneHost 保持一致

97 metallb.universe.tf/loadBalancerIPs: "47.239.98.8"

99adminhost:

100 enabled: true

101 ingress:

102 enabled: true

103 host: admin.gitdata.ai

104 className: nginx

105 tls:

106 enabled: true

107 secretName: ""

108 clusterIssuer: cloudflare-acme-cluster-issuer

109

110webhook:

111 enabled: true

112 ingress:

113 enabled: true

114 host: hooks.gitdata.ai

115 className: nginx

116 tls:

117 enabled: true

118 secretName: ""

119 clusterIssuer: cloudflare-acme-cluster-issuer

120

121# ---------------------------------------------------------------------------

122# AI：评审 / 仓库总结 / release 文案（值对齐 config/local.toml）

123# api_key 走 secret.env -> APP_AI__API_KEY

124# ---------------------------------------------------------------------------

125ai:

126 enabled: true

127 baseUrl: "https://llm-g0x83i22nyj4pub7.cn-beijing.maas.aliyuncs.com/compatible-mode/v1"

128 model: "qwen3.8-flash"

129 reviewEnabled: true

130 summaryEnabled: true

131 releaseNotesEnabled: true

132 queueTopic: "gitdata:ai"

133 dlqTopic: "gitdata:ai:dlq"

134 maxAttempts: 3

135 requestTimeoutSecs: 120

136 contextMaxBytes: 524288

137 maxFindings: 20

138 maxOutputTokens: 4096

139 temperature: "0.2"

140 summaryCommitLimit: 50

141 botUsername: "ai-reviewer"

142 botDisplayName: "AI Reviewer"

143 outputLanguage: "zh-CN"

144

145aihost:

146 enabled: true

147 replicaCount: 1

148 port: 3006

149 concurrency: 2

150 pollIntervalSecs: 5

151 retryBackoffBaseSecs: 2

152 resources:

153 requests: { cpu: 100m, memory: 256Mi }

154 limits: { memory: 1Gi }

155 extraEnv: []

156

157persistence:

158 repos:

159 enabled: true

160 # 预创建的独立 PVC（2Ti, aliyun-nfs-pro, Retain+archiveOnDelete）

161 # 由 kubectl 管理而非 helm，卸载/删除 release 都不会回收

162 existingClaim: gitdata-repos

163 size: 2Ti

164 storageClassName: aliyun-nfs-pro

165 accessModes: ["ReadWriteMany"]

166 mountPath: /opt/gitdata

167

168migrate:

169 enabled: true

170

171synchost:

172 enabled: true

173 replicaCount: 2

174

175mails:

176 enabled: true

177

178im:

179 enabled: true

180

181runner:

182 enabled: false

deploy/README.md +95 −0 View

@@ -0,0 +1,95 @@

1# 部署

3生产部署由 `.gitea/workflows/deploy.yaml` 驱动，推到 `main`/`master` 即自动发布。

5## 前提

7- **Gitea Runner 已配置 KUBECONFIG**（指向 `gitdata` 集群），workflow 不再注入集群凭据。

8- Runner 能访问 `harbor.gitdata.me`（拉取镜像 tag 列表 + 后续 `helm upgrade` 触发的镜像拉取）。

9- 仓库已配置以下 **Repository Secrets**（Settings → Actions → Secrets）：

11| Secret | 用途 |

12| --- | --- |

13| `PROD_SECRET_VALUES` | 生产 values 的 `secret:` 块（见下） |

14| `DOCKER_USERNAME` / `DOCKER_PASSWORD` | Harbor 登录，用于校验镜像已推送 |

15| `DOCKER_REGISTER` | 可选，默认 `harbor.gitdata.me` |

17## 配置拆分：为什么有两个文件

19Helm values 分成两层，避免把凭据写进 git：

21| 文件 | 内容 | 入库 |

22| --- | --- | --- |

23| `deploy/values/prod.yaml` | 域名、副本数、资源配置、AI 模型等**非敏感**项 | ✅ |

24| Gitea Secret `PROD_SECRET_VALUES` | 数据库 URL、SMTP 口令、各签名/加密密钥、Redis 口令、AI API key | ❌ |

26`PROD_SECRET_VALUES` 的完整内容（**19 项**，直接复制粘贴到 Gitea 即可）：

28```yaml

29secret:

30 create: true

31 env:

32 APP_DATABASE__URL: "..."

33 APP_REDIS__URL: "..."

34 APP_CACHE__REDIS__URL: "..."

35 APP_S3__ACCESS_KEY_ID: "..."

36 APP_S3__SECRET_ACCESS_KEY: "..."

37 APP_SEARCH__API_KEY: "..."

38 APP_LIVEKIT__API_KEY: "..."

39 APP_LIVEKIT__API_SECRET: "..."

40 APP_SESSION__KEY: "..."

41 APP_SESSION__DATABASE_KEY: "..."

42 APP_POW__MAC_KEY: "..."

43 APP_AUTH__TOTP_ENCRYPTION_KEY: "..."

44 APP_AUTH__CHAT_JWT_SEC: "..."

45 APP_CI__SECRET_ENCRYPTION_KEY: "..."

46 APP_IMPORT__TOKEN_SECRET: "..."

47 APP_AI__API_KEY: "..."

48 APP_EMAIL__SMTP_PASSWORD: "..."

49 APP_ADMIN__SEED_USERNAME: "..."

50 APP_ADMIN__SEED_PASSWORD: "..."

51```

53实际值在集群运维手上（本机备份：`C:\Users\ZhenYi\k8s-backups\<date>\gitea-secret-values.yaml`）。

54**轮换任何密钥时，只需改 Gitea Secret，不用改仓库。**

56## Workflow 做什么

581. **Resolve image tag** — 默认取本次提交的 `sha-<short>`，可手动指定

592. **Verify image exists** — 轮询 Harbor tags API（最长 ~10min）。

60 `docker-build-push` 与本 workflow 并行触发，不加这步会 `ImagePullBackOff`

613. **Assemble values** — `prod.yaml` + `PROD_SECRET_VALUES` 拼成一份，日志里脱敏

624. **Render** — `helm template` 先渲染，语法/模板错误在这里就暴露

635. **Diff** — 与线上 manifest 逐行对比，评审者可在日志里看到本次真实改动

646. **Deploy** — `helm upgrade --install --atomic --wait --timeout 15m`

65 `--atomic` 保证失败自动回滚到上一版本

667. **Verify rollout** — pod 状态、实际镜像、逐 deployment rollout、Warning 事件

68## 手动操作

70```bash

71# 仅校验不部署

72gitea: workflow_dispatch -> dry_run=true

74# 指定镜像 tag 重发（例如回滚到某个已构建的镜像）

75gitea: workflow_dispatch -> image_tag=sha-xxxxxxx

77# 本地复现 CI 的渲染步骤

78cat deploy/values/prod.yaml > /tmp/values.yaml

79printf '\n' >> /tmp/values.yaml

80cat /path/to/gitea-secret-values.yaml >> /tmp/values.yaml

81helm template gitdata ./chart --namespace gitdata --set image.tag=sha-xxxxxxx -f /tmp/values.yaml

83# 查看当前 release

84helm history gitdata -n gitdata

85```

87## 注意

89- **部署串行**：`concurrency.group = deploy-gitdata` 且 `cancel-in-progress: false`，

90 避免两次 upgrade 互相打断。

91- **迁移 Job**：`migrate.enabled=true` 时作为 pre-upgrade hook 运行，失败会阻断发布。

92- **PVC 不在 chart 管理范围**：`gitdata-repos`（2Ti）由 kubectl 独立管理，

93 `helm uninstall` 不会回收数据。

94- **回滚**：`helm rollback gitdata <revision> -n gitdata`，

95 或用 `image_tag` 重发上一个镜像。