Skip to content

Instantly share code, notes, and snippets.

@life1347
Last active July 22, 2025 06:03
Show Gist options
  • Save life1347/f1940821eebcd659910a966cc0e614f8 to your computer and use it in GitHub Desktop.
Save life1347/f1940821eebcd659910a966cc0e614f8 to your computer and use it in GitHub Desktop.
Upgrade GitLab database from PostgreSQL 14/15 to PostgreSQL 16

升級到 18.2.0-ee 前,查詢升級是否有 breaking change,發現

https://docs.gitlab.com/update/deprecations/#postgresql-14-and-15-no-longer-supported

PostgreSQL 14 & 15 在 18.x 以後不再支援。確認現在 DB 版本

root@gitlab-85f94c4577-b985b:/# postgres --version
postgres (PostgreSQL) 14.17

此時升級資料庫版本之步驟可以按照以下方式 (需使用 17.11.6 版本的 GitLab)

如果是 Kubernetes Pod 啟動的 GitLab,可以透過修改 YAML 後用 exec 進入容器內執行上面 gitlab-ctl 指令

  • 不建議在 GitLab 運作狀態下執行 gitlab-ctl 升級指令
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: gitlab
spec:
  replicas: 1
    spec:
      containers:
      - command:
        - sleep
        args:
        - "36000"
        env:
        ...
        image: life1347/gitlab:17.11.6-ee

首先執行 reconfigure,確保狀況正常

sudo gitlab-ctl reconfigure

如果 gitlab-ctl reconfigure 過程卡在

ruby_block[wait for logrotate service socket] action run

請執行下面指令後,再次 reconfigure

/opt/gitlab/embedded/bin/runsvdir-start &

接著執行

sudo gitlab-ctl pg-upgrade -V 16

執行完畢後見到下列訊息表示成功升級至 PostgreSQL 16

Please verify everything is working and run the following if so
sudo rm -rf /var/opt/gitlab/postgresql/data.14
sudo rm -f /var/opt/gitlab/postgresql-version.old

接著,請修改 Deployment 使用 GitLab 官方 Arm64 Image 即可正常啟動

如果遇到無法啟動,


{"method":"POST","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"create","status":500,"time":"2025-07-21T15:51:10.036Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"otp_attempt":"[FILTERED]","remember_me":"0"}}],"correlation_id":"01K0PTX191T983FDPZCW07Q2KQ","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.organization_id":1,"remote_ip":"1.34.53.56","user_id":1,"username":"root","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36","queue_duration_s":0.00877,"request_urgency":"low","target_duration_s":5,"cf_ray":"962bdba7b9a0f640-LAX","cf_ipcountry":"TW","redis_calls":8,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.002164,"redis_read_bytes":1496,"redis_write_bytes":782,"redis_feature_flag_calls":2,"redis_feature_flag_duration_s":0.000357,"redis_feature_flag_read_bytes":2,"redis_feature_flag_write_bytes":248,"redis_sessions_calls":5,"redis_sessions_allowed_cross_slot_calls":1,"redis_sessions_duration_s":0.00162,"redis_sessions_read_bytes":1492,"redis_sessions_write_bytes":386,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000187,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":148,"db_count":11,"db_write_count":2,"db_cached_count":1,"db_txn_count":3,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_replica_count":0,"db_primary_count":11,"db_replica_write_count":0,"db_primary_write_count":2,"db_replica_cached_count":0,"db_primary_cached_count":1,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.024,"db_main_txn_count":3,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_main_count":11,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_main_write_count":2,"db_ci_write_count":0,"db_main_replica_write_count":0,"db_ci_replica_write_count":0,"db_main_cached_count":1,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_main_txn_max_duration_s":0.071,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_main_txn_duration_s":0.081,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_main_duration_s":0.024,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"path_traversal_check_duration_s":0.000032,"cpu_s":1.143745,"mem_objects":92828,"mem_bytes":8861560,"mem_mallocs":49875,"mem_total_bytes":12574680,"pid":707,"worker_id":"puma_0","rate_limiting_gates":[],"exception.class":"NoMethodError","exception.message":"undefined method `markdown_maintain_indentation' for #<UserPreference id: 1, user_id: 1, issue_notes_filter: 0, merge_request_notes_filter: 0, created_at: \"2023-08-16 05:11:37.939831000 +0000\", updated_at: \"2023-08-16 05:11:37.939831000 +0000\", epics_sort: nil, roadmap_epics_state: nil, epic_notes_filter: 0, issues_sort: nil, merge_requests_sort: nil, roadmaps_sort: nil, first_day_of_week: nil, timezone: nil, time_display_relative: true, projects_sort: nil, show_whitespace_in_diffs: true, sourcegraph_enabled: nil, setup_for_company: nil, render_whitespace_in_code: false, tab_width: 8, view_diffs_file_by_file: false, gitpod_enabled: false, markdown_surround_selection: true, diffs_deletion_color: nil, diffs_addition_color: nil, markdown_automatic_lists: true, use_new_navigation: nil, achievements_enabled: true, pinned_nav_items: {}, pass_user_identities_to_ci_jwt: [FILTERED], enabled_following: true, visibility_pipeline_id_type: \"id\", project_shortcut_buttons: true, enabled_zoekt: true, keyboard_shortcuts_enabled: true, time_display_format: 0, home_organization_id: nil, extensions_marketplace_opt_in_status: \"unset\", early_access_program_participant: false, early_access_program_tracking: false, organization_groups_projects_sort: nil, organization_groups_projects_display: \"projects\", use_work_items_view: false, dpop_enabled: false, text_editor_type: \"not_set\", extensions_marketplace_opt_in_url: nil, merge_request_dashboard_list_type: \"action_based\", dark_color_scheme_id: 2>","exception.backtrace":["activemodel (7.1.5.1) lib/active_model/attribute_methods.rb:511:in `method_missing'","gems/activerecord-gitlab/lib/active_record/gitlab_patches/attribute_methods.rb:32:in `method_missing'","app/models/user.rb:427:in `markdown_maintain_indentation'","lib/gitlab/gon_helper.rb:20:in `add_gon_variables'","ee/lib/ee/gitlab/gon_helper.rb:10:in `add_gon_variables'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:403:in `block in make_lambda'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:183:in `block (2 levels) in halting_and_conditional'","actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:184:in `block in halting_and_conditional'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `block in invoke_before'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `each'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:598:in `invoke_before'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:119:in `block in run_callbacks'","lib/gitlab/ip_address_state.rb:11:in `with'","app/controllers/application_controller.rb:469:in `set_current_ip_address'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'","sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'","sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span'","sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span'","sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:141:in `run_callbacks'","actionpack (7.1.5.1) lib/abstract_controller/callbacks.rb:258:in `process_action'","actionpack (7.1.5.1) lib/action_controller/metal/rescue.rb:25:in `process_action'","actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'","activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.1.5.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument'","activesupport (7.1.5.1) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.1.5.1) lib/action_controller/metal/instrumentation.rb:73:in `process_action'","actionpack (7.1.5.1) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'","activerecord (7.1.5.1) lib/active_record/railties/controller_runtime.rb:32:in `process_action'","actionpack (7.1.5.1) lib/abstract_controller/base.rb:160:in `process'","actionview (7.1.5.1) lib/action_view/rendering.rb:40:in `process'","actionpack (7.1.5.1) lib/action_controller/metal.rb:227:in `dispatch'","actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `block in dispatch'","lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'","actionpack (7.1.5.1) lib/action_controller/metal.rb:307:in `dispatch'","actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.1.5.1) lib/action_dispatch/routing/mapper.rb:30:in `block in <class:Constraints>'","actionpack (7.1.5.1) lib/action_dispatch/routing/mapper.rb:60:in `serve'","actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:51:in `block in serve'","config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'","config/initializers/action_dispatch_journey_router.rb:25:in `map!'","config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'","actionpack (7.1.5.1) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.1.5.1) lib/action_dispatch/routing/route_set.rb:882:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","omniauth (2.1.3) lib/omniauth/strategy.rb:472:in `call_app!'","omniauth-saml (2.2.3) lib/omniauth/strategies/saml.rb:86:in `other_phase'","omniauth (2.1.3) lib/omniauth/strategy.rb:195:in `call!'","omniauth (2.1.3) lib/omniauth/strategy.rb:169:in `call'","flipper (0.28.3) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.28.3) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:83:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/ci/job_token/middleware.rb:11:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'","rack (2.2.17) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.17) lib/rack/etag.rb:27:in `call'","rack (2.2.17) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.17) lib/rack/head.rb:12:in `call'","actionpack (7.1.5.1) lib/action_dispatch/http/permissions_policy.rb:36:in `call'","actionpack (7.1.5.1) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.17) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.17) lib/rack/session/abstract/id.rb:260:in `call'","lib/gitlab/middleware/secure_headers.rb:11:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/cookies.rb:689:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'","activesupport (7.1.5.1) lib/active_support/callbacks.rb:101:in `run_callbacks'","actionpack (7.1.5.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'","sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'","lib/gitlab/middleware/path_traversal_check.rb:40:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'","sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'","sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'","sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'","sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'","sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'","sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `block in call'","activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `block in tagged'","activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:39:in `tagged'","activesupport (7.1.5.1) lib/active_support/tagged_logging.rb:139:in `tagged'","activesupport (7.1.5.1) lib/active_support/broadcast_logger.rb:241:in `method_missing'","railties (7.1.5.1) lib/rails/rack/logger.rb:24:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'","rack (2.2.17) lib/rack/method_override.rb:24:in `call'","rack (2.2.17) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'","rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:83:in `call'","gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.39.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.39.0) lib/labkit/middleware/rack.rb:21:in `call'","rack (2.2.17) lib/rack/sendfile.rb:110:in `call'","actionpack (7.1.5.1) lib/action_dispatch/middleware/request_id.rb:28:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","railties (7.1.5.1) lib/rails/engine.rb:536:in `call'","railties (7.1.5.1) lib/rails/railtie.rb:226:in `public_send'","railties (7.1.5.1) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:12:in `call'","rack (2.2.17) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.17) lib/rack/urlmap.rb:58:in `each'","rack (2.2.17) lib/rack/urlmap.rb:58:in `call'","puma (6.6.0) lib/puma/configuration.rb:279:in `call'","puma (6.6.0) lib/puma/request.rb:99:in `block in handle_request'","puma (6.6.0) lib/puma/thread_pool.rb:390:in `with_force_shutdown'","puma (6.6.0) lib/puma/request.rb:98:in `handle_request'","puma (6.6.0) lib/puma/server.rb:472:in `process_client'","puma (6.6.0) lib/puma/server.rb:254:in `block in run'","puma (6.6.0) lib/puma/thread_pool.rb:167:in `block in spawn_thread'"],"db_duration_s":0.0606,"view_duration_s":0.0,"duration_s":1.18921}

lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'

主要錯誤原因是

..."exception.class":"NoMethodError","exception.message":"undefined method `markdown_maintain_indentation' for #<UserPreference id: 1, user_id: 1, issue_notes_filter: 0,...

以我遇到的狀況是 GitLab 資料庫 Schema 沒有正確遷移,可以 gitlab-rake db:migrate:status 確認

$ gitlab-rake db:migrate:status

...
   up     20250404035239  post     17.11        RequeueBackfillArchivedAndTraversalIdsToVulnerabilityStatistics
  down    20250408160608  post     17.11        IndexSprintsWithoutGroup
  down    20250408170948  post     17.11        AddIndexToResourceIterationEventsNamespaceId
  down    20250408171252  post     17.11        AddResourceIterationEventsNamespaceIdFk
  down    20250408172437  post     17.11        DropTempIdxResourceIterationEventsWithoutIteration
  down    20250408172501  post     17.11        DropTempIdxSprintsWithoutGroupId
   up     20250409031624  post     17.11        TrackRecordDeletionsOfLfsObjects
  down    20250409172701  post     17.11        QueueBackfillResourceIterationEventsNamespaceId
   up     20250409214157  post     17.11        RemoveTopicsOrganizationIdDefault
  down    20250409231753  post     17.11        FinalizeHkBackfillIssuableSeveritiesNamespaceId
   up     20250410024159  post     17.11        AddIncidentManagementIssuableEscalationStatusesNamespaceIdNotNull
   up     20250410130252  post     17.11        FinalizeBackfillFinishOnboardingForEnterpriseUser
  down    20250410213433  post     17.11        AddPackagesNugetMetadataProjectIdNotNull
  down    20250411043427  post     17.11        AddTempIndexOnNotesForProjectsNullAndId
  down    20250312124044  regular  18.0         ChangeGeoConcurrencyDefaultSettings
  down    20250410083733  regular  18.0         AddPreApprovedAgentPrivilegesToDuoWorkflowsWorkflows
...

出現 down 表示該 Schema 沒有遷移成功,需要手動觸發

$ gitlab-rake db:migrate

main: == 20250711212414 AddIndexToAiCatalogItemsDeletedAt: migrating ================
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0011s
...
main: == 20250624140120 DropPCiPipelinesConfig: migrating ===========================
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- drop_table(:p_ci_pipelines_config)
main:    -> 0.0070s
main: == 20250624140120 DropPCiPipelinesConfig: migrated (0.0186s) ==================

main: == [advisory_lock_connection] object_id: 53120, pg_backend_pid: 1087

結束後,最後確認都變為 up 即可

$ gitlab-rake db:migrate:status

...
   up     20250713232042  post     18.2         FinalizeHkBackfillOnboardingStatusSetupForCompany
   up     20250713233817  post     18.2         FinalizeHkBackfillBulkImportFailuresProjectId
   up     20250705081114  regular  18.3         AddSdrsConfiguration
   up     20250711211826  regular  18.3         AddDeletedAtToAiCatalogItems
   up     20250711212414  regular  18.3         AddIndexToAiCatalogItemsDeletedAt
   up     20250624133912  post     18.3         RemoveCiPipelinesConfigPipelineForeignKey
   up     20250624140120  post     18.3         DropPCiPipelinesConfig
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment