在 Rancher 中升级容器时，出现数据库迁移重复执行，进而报错的排查分析
1、在 Rancher 中第一次升级容器时，出现数据库迁移重复执行，进而报错升级失败。如图1
图1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | *** applying m210106_122427_add_columns_to_task_table> add column article_type_id integer NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` to table {{%task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_type_id'The SQL being executed was: ALTER TABLE `cpa_task` ADD `article_type_id` int(11) NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii\db\Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii\db\Command->internalExecute('ALTER TABLE `cp...')#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii\db\Command->execute()#3 /mcloud/www/ccp_api/console/migrations/m210106_122427_add_columns_to_task_table.php(15): yii\db\Migration->addColumn('{{%task}}', 'article_type_id', Object(yii\db\mysql\ColumnSchemaBuilder))#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m210106_122427_add_columns_to_task_table->safeUp()#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii\db\Migration->up()#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii\console\controllers\BaseMigrateController->migrateUp('m210106_122427_...')#7 [internal function]: yii\console\controllers\BaseMigrateController->actionUp(0)#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\InlineAction->runWithParams(Array)#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii\base\Controller->runAction('', Array)#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii\console\Controller->runAction('', Array)#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii\base\Module->runAction('migrate', Array)#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii\console\Application->runAction('migrate', Array)#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii\console\Application->handleRequest(Object(yii\console\Request))#15 /mcloud/www/ccp_api/yii(23): yii\base\Application->run()#16 {main}*** failed to apply m210106_122427_add_columns_to_task_table (time: 0.006s) |
2、已经确定数据库结构并未被手动修改过。分析容器的健康检查。周期性向容器发出请求，以检测其健康状态。默认设置下，readiness和liveness使用相同的配置参数。对于应用初始化较长的容器，需要增加就绪检测时间。初步分析结论：由于数据库迁移执行时间过长，导致容器初始化时间过长，导致健康检查失败，再导致容器不断启动运行，数据库迁移重复执行。决定增加就绪检测时间。运行多少时间后开始检测：10秒 -> 100秒。如图2
图2
3、在初始部署时仍然报错：ReplicaSet “api-77dfcc7cfb” has timed out progressing.; Deployment does not have minimum availability.。如图3
图3
4、查看日志，仍然是数据库迁移重复执行的问题。查看 MySQL 中的表结构与迁移表中的记录，确定迁移记录：m210402_052030_add_columns_to_channel_app_source_table 在迁移表中不存在，但是 MySQL 中的表结构已经是此迁移记录执行后的结果。如图4
图4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | m210402_052030_add_columns_to_channel_app_source_table*** applying m201105_020453_add_article_read_count_column_to_channel_app_task_table> add column article_read_count integer NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` to table {{%channel_app_task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_read_count'The SQL being executed was: ALTER TABLE `cpa_channel_app_task` ADD `article_read_count` int(11) NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii\db\Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii\db\Command->internalExecute('ALTER TABLE `cp...')#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii\db\Command->execute()#3 /mcloud/www/ccp_api/console/migrations/m201105_020453_add_article_read_count_column_to_channel_app_task_table.php(15): yii\db\Migration->addColumn('{{%channel_app_...', 'article_read_co...', Object(yii\db\mysql\ColumnSchemaBuilder))#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m201105_020453_add_article_read_count_column_to_channel_app_task_table->safeUp()#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii\db\Migration->up()#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii\console\controllers\BaseMigrateController->migrateUp('m201105_020453_...')#7 [internal function]: yii\console\controllers\BaseMigrateController->actionUp(0)#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\InlineAction->runWithParams(Array)#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii\base\Controller->runAction('', Array)#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii\console\Controller->runAction('', Array)#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii\base\Module->runAction('migrate', Array)#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii\console\Application->runAction('migrate', Array)#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii\console\Application->handleRequest(Object(yii\console\Request))#15 /mcloud/www/ccp_api/yii(23): yii\base\Application->run()#16 {main}*** failed to apply m201105_020453_add_article_read_count_column_to_channel_app_task_table (time: 0.006s)0 from 27 migrations were applied.Migration failed. The rest of the migrations are canceled. |
5、运行多少时间后开始检测。：10秒 -> 100秒。如图5
图5
近期评论