基于企鹅号的视频文件分片上传的实现流程，包含队列、文件切片、while 循环等
1、数据库结构的设计，一张资源表，一张企鹅号的视频文件分片上传表，一张企鹅号的事务表，结构如下：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | 28、asset：资源 Assetid 主键channel_id 渠道IDchannel_code 渠道代码，qq：企鹅号；wx：微信公众帐号channel_type_id 渠道的类型IDchannel_type_code 渠道的类型代码，qq_cw：企鹅号的内容网站应用；qq_tp：企鹅号的第三方服务平台应用；wx：微信公众帐号应用source 来源，xContent：内容库；vms：视频管理系统；cms：内容管理系统；spider：自媒体type 资源文件的类型，image：图片；video：视频absolute_url 来源的资源文件的绝对URLrelative_path 渠道发布的资源文件的相对路径size 文件大小，单位(字节)task_id 任务IDchannel_article_id 渠道的文章IDstatus 状态，0：禁用；1：启用is_deleted 是否被删除，0：否；1：是created_at 创建时间updated_at 更新时间deleted_at 删除时间 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | 29、qq_video_multipart_upload：企鹅号的视频文件分片上传 QqCwVideoMultipartUploadid 主键asset_id 资源IDqq_app_task_id 企鹅号的应用的任务IDqq_app_id 企鹅号的应用IDqq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用size 视频文件大小，单位(字节)md5 视频文件MD5值sha 视频文件SHA-1值transaction_id 上传的唯一事务IDmediatrunk 视频 mediatrunk 文件start_offset 分片的起始位置(从0开始计数)end_offset 分片的结束位置vid 视频文件唯一标示IDstatus 状态，0：禁用；1：待上传；2：上传中；3：上传中(已失败)；4：已上传is_deleted 是否被删除，0：否；1：是created_at 创建时间updated_at 更新时间deleted_at 删除时间 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | 30、qq_transaction：企鹅号的事务 QqTransactionid 主键group_id 租户IDqq_app_task_id 企鹅号的应用的任务IDqq_app_id 企鹅号的应用IDqq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用qq_article_id 企鹅号的文章IDqq_video_multipart_upload_id 企鹅号的应用的视频文件分片上传IDtransaction_id 事务IDtype 类型，1：文章；2：视频transaction_ctime 事务创建时间ext_err 扩展的错误transaction_err_msg 事务的错误信息article_abstract 文章摘要article_type 文章类型，取值：普通文章，图文文章，视频文章，直播文章，RTMP直播文章article_type_code 文章类型代码，normal：文章；multivideos：视频；images：组图；live：直播article_url 文章快报链接article_imgurl 文章封面图article_title 文章标题article_pub_flag 文章发布状态，取值：未发布，发布成功，审核中article_pub_time 文章发布时间article_video_title 视频文章标题article_video_desc 视频文章描述article_video_type 视频文章类型，视频article_video_vid 视频文章的视频唯一IDtask_id 任务IDstatus 状态，0：禁用；1：成功；2：失败；3：处理中is_deleted 是否被删除，0：否；1：是created_at 创建时间updated_at 更新时间deleted_at 删除时间 |
2、资源的上传是基于队列实现的，因此会先将资源数据存储至资源表，进而入上传资源的队列，再执行上传资源的作业
3、执行 1 次接口请求，此时资源表已经存在资源的相应数据，且入复制资源的队列
1 2 3 4 5 6 | PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0Jobs- waiting: 1- delayed: 0- reserved: 0- done: 0 |
4、执行复制资源队列中的任务命令，会复制相应的资源，且将资源的相对路径存储至资源表中
1 2 3 4 5 | PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=02018-11-15 10:05:56 [pid: 23112] - Worker is started2018-11-15 10:05:57 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Started2018-11-15 10:06:08 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Done (11.394 s)2018-11-15 10:06:08 [pid: 23112] - Worker is stopped (0:00:12) |
5、复制资源队列中的任务执行成功后，会入上传资源的队列
1 2 3 4 5 6 | PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/info --color=0Jobs- waiting: 1- delayed: 0- reserved: 0- done: 0 |
6、上传资源的代码分为 2 个部分，一为文件切片，需要将视频资源文件切片为100M大小的小文件，\channel-pub-api\common\services\AssetService.php
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | /** * 文件切片 * @param string $fileAbsolutePath 需要切片的文件的绝对路径 * 格式如下：E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件.mp4 * * @param int $size 104857600，单位为字节 * * 生成文件列表如下： * E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_0.mp4 * E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_1.mp4 */public static function cut($fileAbsolutePath, $size){ // 获取需要切片的文件的路径信息 $pathInfo = pathinfo($fileAbsolutePath); $i = 0; $handle = fopen($fileAbsolutePath, "rb"); while (!feof($handle)) { $cutHandle = fopen($pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'], "wb"); fwrite($cutHandle, fread($handle, $size)); fclose($cutHandle); unset($cutHandle); $i++; } fclose($handle);} |
7、HTTP请求，企鹅号的内容网站应用的视频文件分片上传，\channel-pub-api\common\logics\http\qq_api\Video.php
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | <?php/** * Created by PhpStorm. * User: Qiang Wang * Date: 2018/10/26 * Time: 15:33 */namespace common\logics\http\qq_api;use Yii;use yii\httpclient\Client;use yii\web\ServerErrorHttpException;/** * 企鹅号接口的视频 * * @author Qiang Wang <shuijingwanwq@163.com> * @since 1.0 */class Video extends Model{ const CUT_SIZE = 104857600; //视频分片上传的切片大小：100M /** * HTTP请求，申请企鹅号的内容网站应用的视频文件分片上传的唯一事务ID * * @param array $data 数据 * 格式如下： * [ * 'accessToken' => 'KCK0TIV8NDY44ONAEFI2QW', // 企鹅平台企鹅号应用授权调用凭据 * 'size' => 9135849, // 视频文件大小，单位(字节) * 'md5' => 'd081c619ef7dc62eb2aa29c7c83c6f26', // 视频文件MD5值 * 'sha' => '28a1076b867bed8202df5b3f656b798148e58ced', // 视频文件SHA-1值 * ] * * @return array|false * 格式如下： * 企鹅号的内容网站应用的视频文件分片上传的唯一事务ID * [ * 'message' => '', // 说明 * 'data' => [ // 数据 * 'transaction_id' => '780930255958621794', // 上传的唯一事务ID * ], * ] * * 失败(将错误保存在 [[yii\base\Model::errors]] 属性中) * false * * @throws ServerErrorHttpException 如果响应状态码不等于20x */ public function clientUploadReady($data) { $response = Yii::$app->qqApiHttps->createRequest() ->setMethod('post') ->setUrl('video/clientuploadready') ->setData([ 'access_token' => $data['accessToken'], 'size' => $data['size'], 'md5' => $data['md5'], 'sha' => $data['sha'], ]) ->send(); // file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-ready_' . $data['size'] . '_' . time() . '.txt', $response->data['data']['transaction_id']); // 检查响应状态码是否等于20x if ($response->isOk) { // 检查业务逻辑是否成功 if ($response->data['code'] === 0) { $responseData = ['message' => '', 'data' => $response->data['data']]; return $responseData; } else { $this->addError('id', $response->data['msg']); return false; } } else { throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041); } } /** * HTTP请求，企鹅号的内容网站应用的视频文件分片上传 * * @param array $data 数据 * 格式如下： * [ * 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据 * 'transactionId' => '780930287703152921', // 上传的唯一事务ID * 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件 * 'startOffset' => 0, // 分片的起始位置(从0开始计数) * ] * * @return array|false * 格式如下： * 企鹅号的内容网站应用的视频文件分片上传 * [ * 'message' => '', // 说明 * 'data' => [ // 数据 * 'end_offset' => 2198151, // 分片的结束位置 * 'start_offset' => 2198151, // 分片的起始位置 * 'transaction_id' => 780930255958621794, // 上传的唯一事务ID * ], * ] * * 失败(将错误保存在 [[yii\base\Model::errors]] 属性中) * false * * @throws ServerErrorHttpException 如果响应状态码不等于20x */ public function clientUploadTrunk($data) { $response = Yii::$app->qqApiHttp->createRequest() ->setMethod('post') ->setUrl('video/clientuploadtrunk?access_token=' . $data['accessToken'] . '&transaction_id=' . $data['transactionId'] . '&start_offset=' . $data['startOffset']) ->addFile('mediatrunk', $data['mediatrunk']) ->send(); // file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-trunk_' . $data['transactionId'] . '_' . $data['startOffset'] . '_' . time() . '.txt', $response->data['code']); // 检查响应状态码是否等于20x if ($response->isOk) { // 检查业务逻辑是否成功 if ($response->data['code'] === 0) { $responseData = ['message' => '', 'data' => $response->data['data']]; return $responseData; } elseif ($response->data['code'] === 40027) { // 无效的事务ID $this->addError('id', $response->data['code']); return false; } else { $this->addError('id', $response->data['msg']); return false; } } else { throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041); } }} |
8、上传视频文件的代码，\channel-pub-api\common\services\QqCwVideoMultipartUploadService.php
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | /** * HTTP请求，企鹅号的内容网站应用的视频文件分片上传 * @param array $data 数据 * 格式如下： * * [ * 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据 * 'transactionId' => '780930287703152921', // 上传的唯一事务ID * 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件 * 'startOffset' => 0, // 分片的起始位置(从0开始计数) * ] * * @return array * 格式如下： * * [ * 'end_offset' => 2198151, // 分片的结束位置 * 'start_offset' => 2198151, // 分片的起始位置 * 'transaction_id' => 780930255958621794, // 上传的唯一事务ID * ] * * @throws ServerErrorHttpException */public function httpUploadTrunk($data){ /* HTTP请求，企鹅号的内容网站应用的视频文件分片上传 */ $httpQqApiVideo = new HttpQqApiVideo(); $uploadTrunk = $httpQqApiVideo->clientUploadTrunk($data); if ($uploadTrunk === false) { if ($httpQqApiVideo->hasErrors()) { foreach ($httpQqApiVideo->getFirstErrors() as $message) { $firstErrors = $message; break; } throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35042'), ['firstErrors' => $firstErrors])), 35042); } elseif (!$httpQqApiVideo->hasErrors()) { throw new ServerErrorHttpException('Penguin\'s content website application api HTTP requests fail for unknown reasons.'); } } return $uploadTrunk['data'];}/** * 企鹅号的内容网站应用的视频文件分片上传 * * @param int $assetId 资源ID * 格式如下：1 * * @param int $qqCwAppTaskId 企鹅号的内容网站应用的任务ID * 格式如下：6 * * @throws ServerErrorHttpException * @throws \Throwable */public function upload($assetId, $qqCwAppTaskId){ // 基于ID查找状态为启用的单个数据模型(资源) $assetEnabledItem = AssetService::findModelEnabledById($assetId); // 基于ID查找状态为启用的单个数据模型(任务) $taskEnabledItem = TaskService::findModelEnabledById($assetEnabledItem->task_id); // 基于ID查找状态为发布中的单个数据模型(企鹅号的内容网站应用的任务) $qqCwAppTaskPublishItem = QqCwAppTaskService::findModelPublishById($qqCwAppTaskId); // 基于ID查找状态为启用的单个数据模型(企鹅号的内容网站应用) $qqCwAppEnabledItem = QqCwAppService::findModelEnabledById($qqCwAppTaskPublishItem->qq_cw_app_id); // 基于企鹅号的内容网站应用ID获取有效的 Access Token $qqCwAccessTokenService = new QqCwAccessTokenService(); $accessTokenValidity = $qqCwAccessTokenService->getAccessTokenValidityByQqCwAppId($qqCwAppEnabledItem->id); // 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型 $qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id); $data = [ 'assetId' => $qqVideoMultipartUploadItem->asset_id, 'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id, 'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id, 'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type, 'size' => $qqVideoMultipartUploadItem->size, 'md5' => $qqVideoMultipartUploadItem->md5, 'sha' => $qqVideoMultipartUploadItem->sha, 'transactionId' => (string) $qqVideoMultipartUploadItem->transaction_id, 'mediatrunk' => $qqVideoMultipartUploadItem->mediatrunk, 'startOffset' => $qqVideoMultipartUploadItem->start_offset, 'endOffset' => $qqVideoMultipartUploadItem->end_offset, 'vid' => $qqVideoMultipartUploadItem->vid, 'status' => $qqVideoMultipartUploadItem->status, ]; // 文件切片 AssetService::cut(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk, HttpQqApiVideo::CUT_SIZE); // 获取需要切片的文件的路径信息 $pathInfo = pathinfo(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk); // 判断分片的起始位置与分片的结束位置是否等于视频文件大小，如果相等则中断分片上传，否则继续执行分片上传 $i = 0; while ($qqVideoMultipartUploadItem->start_offset != $qqVideoMultipartUploadItem->size && $qqVideoMultipartUploadItem->end_offset != $qqVideoMultipartUploadItem->size) { // file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/while_' . $assetId . '_' . $qqCwAppId . '_' . $qqVideoMultipartUploadItem->start_offset . '_' . $qqVideoMultipartUploadItem->end_offset . '_' . time() . '.txt', $assetId); // HTTP请求，企鹅号的内容网站应用的视频文件分片上传 $httpUploadTrunkData = [ 'accessToken' => $accessTokenValidity->access_token, 'transactionId' => $qqVideoMultipartUploadItem->transaction_id, 'mediatrunk' => $pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'], 'startOffset' => $qqVideoMultipartUploadItem->start_offset, ]; $uploadTrunkData = $this->httpUploadTrunk($httpUploadTrunkData); // 基于资源ID、企鹅号的内容网站应用ID插入/更新数据至企鹅号的内容网站应用的视频文件分片上传 $data['startOffset'] = $uploadTrunkData['start_offset']; $data['endOffset'] = $uploadTrunkData['end_offset']; if ($uploadTrunkData['start_offset'] != $qqVideoMultipartUploadItem->size && $uploadTrunkData['end_offset'] != $qqVideoMultipartUploadItem->size) { $data['status'] = QqVideoMultipartUpload::STATUS_UPLOADING; } else { // HTTP请求，基于上传的唯一事务ID获取事务信息 $qqTransactionService = new QqTransactionService(); $qqTransactionServiceHttpTransactionInfoData = [ 'accessToken' => $accessTokenValidity->access_token, 'transactionId' => $qqVideoMultipartUploadItem->transaction_id, ]; $qqTransactionServiceHttpTransactionInfoResult = $qqTransactionService->httpTransactionInfo($qqTransactionServiceHttpTransactionInfoData); // 创建企鹅号的事务 $qqTransactionServiceCreateData = [ 'groupId' => $taskEnabledItem->group_id, 'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id, 'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id, 'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type, 'qqArticleId' => 0, 'qqVideoMultipartUploadId' => $qqVideoMultipartUploadItem->id, 'transactionId' => $qqVideoMultipartUploadItem->transaction_id, 'type' => $qqTransactionService::getTypeByHttpQqApiTransactionType($qqTransactionServiceHttpTransactionInfoResult['transaction_type']), 'transactionCtime' => $qqTransactionServiceHttpTransactionInfoResult['transaction_ctime'], 'extErr' => $qqTransactionServiceHttpTransactionInfoResult['ext_err'], 'transactionErrMsg' => $qqTransactionServiceHttpTransactionInfoResult['transaction_err_msg'], 'articleAbstract' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_abstract'], 'articleType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_type'], 'articleTypeCode' => '', 'articleUrl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_url'], 'articleImgurl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_imgurl'], 'articleTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_title'], 'articlePubFlag' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_flag'], 'articlePubTime' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_time'], 'articleVideoTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['title'], 'articleVideoDesc' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['desc'], 'articleVideoType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['type'], 'articleVideoVid' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['vid'], 'taskId' => $assetEnabledItem->task_id, 'status' => QqTransaction::STATUS_PROCESSING, ]; $qqTransactionServiceCreateResult = $qqTransactionService->create($qqTransactionServiceCreateData); if ($qqTransactionServiceCreateResult['status'] === false) { throw new ServerErrorHttpException($qqTransactionServiceCreateResult['message'], $qqTransactionServiceCreateResult['code']); } $data['status'] = QqVideoMultipartUpload::STATUS_UPLOADED; } $result = $this->saveModelByData($data); if ($result['status'] === false) { throw new ServerErrorHttpException($result['message'], $result['code']); } // 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型 $qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id); $i++; }} |
9、执行上传资源队列中的任务命令，会切片文件，上传相应的资源，且更新企鹅号的视频文件分片上传表、新增企鹅号的事务
1 2 3 4 5 | PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/run --verbose=1 --isolate=1 --color=02018-11-15 11:02:29 [pid: 39632] - Worker is started2018-11-15 11:02:30 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Started2018-11-15 11:06:48 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Done (257.816 s)2018-11-15 11:06:48 [pid: 39632] - Worker is stopped (0:04:19) |
10、查看生成的切片小文件，由于切片大小为100M，396 MB (415,352,401 字节)的文件切片为4个小的文件，如图1
图1
11、上传成功后，分片的起始位置与分片的结束位置皆等于文件的大小：415352401，如图2
图2
12、在企鹅号后台查看我的素材，视频文件已经分片上传成功，如图3
图3
近期评论