在 K8s 中 HTTP 响应 502,message:An invalid response was received from the upstream server
1、在 K8s 中 HTTP 响应 502 Bad Gateway,message:An invalid response was received from the upstream server。从上游服务器收到无效响应。502 Bad Gateway 错误具体表示服务器收到来自入站服务器的无效响应。kong/2.5.0。如图1
2、原因在于 GET 请求参数过多,参数量过大。参数总字数超过 1300 个。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | {{host_info}}{{base_url}}/v{{module}}/task-group-channel-app-sources?filter[group_id]={{group_id}}&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&filter[channel_type_code][in][19]=vtt_cashtoutiao&filter[channel_type_code][in][20]=vtt_dianping&filter[channel_type_code][in][21]=vtt_weibo&filter[channel_type_code][in][22]=vtt_sina&filter[channel_type_code][in][23]=vtt_iqiyi&filter[channel_type_code][in][24]=vtt_toutiao_pro&filter[channel_type_code][in][25]=vtt_bilibili&filter[channel_type_code][in][26]=vtt_netease&filter[channel_type_code][in][27]=vtt_sogou&per-page=100&group_id={{group_id}} filter[channel_type_code][in][0]:vtt_douyin filter[channel_type_code][in][1]:vtt_douyin_pro filter[channel_type_code][in][2]:vtt_kuaishou filter[channel_type_code][in][3]:vtt_kuaishou_pro filter[channel_type_code][in][4]:vtt_baijia filter[channel_type_code][in][5]:vtt_baijia_pro filter[channel_type_code][in][6]:vtt_dayu filter[channel_type_code][in][7]:vtt_dayu_pro filter[channel_type_code][in][8]:vtt_qq filter[channel_type_code][in][9]:vtt_yidian filter[channel_type_code][in][10]:vtt_qutoutiao filter[channel_type_code][in][11]:vtt_miaopai filter[channel_type_code][in][12]:vtt_meipai filter[channel_type_code][in][13]:vtt_rrtv filter[channel_type_code][in][14]:vtt_sohutv filter[channel_type_code][in][15]:vtt_sohu filter[channel_type_code][in][16]:vtt_acfun filter[channel_type_code][in][17]:vtt_yinyuetai filter[channel_type_code][in][18]:vtt_neteasemusic filter[channel_type_code][in][19]:vtt_cashtoutiao filter[channel_type_code][in][20]:vtt_dianping filter[channel_type_code][in][21]:vtt_weibo filter[channel_type_code][in][22]:vtt_sina filter[channel_type_code][in][23]:vtt_iqiyi filter[channel_type_code][in][24]:vtt_toutiao_pro filter[channel_type_code][in][25]:vtt_bilibili filter[channel_type_code][in][26]:vtt_netease filter[channel_type_code][in][27]:vtt_sogou |
3、总计 28 个请求参数,删减为 14 个。响应 200。如图2
4、查看容器中的 Nginx 日志,确定接口响应 502 时候,实际是响应 200 的。如图3
1 | {"environment-info":{"pod-id":"api-7cc8cc4c9c-hx89x","namespace":"ccp","pod-ip":"10.42.4.155","pod-hostip":"192.168.3.14"},"request":{"headers":{"referer":"-","host":"api.ccp","x-request-id":"156f0a80cc88c1ee0bb79b1f06d777f1","x-forwarded-for":"101.207.139.251, 10.42.0.231","user-agent":"PostmanRuntime/7.28.4"},"method":"GET","uri":"/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&filter[channel_type_code][in][19]=vtt_cashtoutiao&filter[channel_type_code][in][20]=vtt_dianping&filter[channel_type_code][in][21]=vtt_weibo&filter[channel_type_code][in][22]=vtt_sina&filter[channel_type_code][in][23]=vtt_iqiyi&filter[channel_type_code][in][24]=vtt_toutiao_pro&filter[channel_type_code][in][25]=vtt_bilibili&filter[channel_type_code][in][26]=vtt_netease&filter[channel_type_code][in][27]=vtt_sogou&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753","url":"http://api.ccp/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&filter[channel_type_code][in][19]=vtt_cashtoutiao&filter[channel_type_code][in][20]=vtt_dianping&filter[channel_type_code][in][21]=vtt_weibo&filter[channel_type_code][in][22]=vtt_sina&filter[channel_type_code][in][23]=vtt_iqiyi&filter[channel_type_code][in][24]=vtt_toutiao_pro&filter[channel_type_code][in][25]=vtt_bilibili&filter[channel_type_code][in][26]=vtt_netease&filter[channel_type_code][in][27]=vtt_sogou&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753","@timestamp":"2021-09-30T13:28:24+08:00","upstreamaddr":"127.0.0.1:9000","response":{"responsetime":"0.430","upstream-response-time":"0.430"},"size":"0","status":"200"},"module-name":"default_ccp_api"} |
5、决定在另外一台容器中调用此接口,跳过 Kong 网关。报错:curl: (3) [globbing] error: bad range specification after pos 60。如图4
1 | curl "http://api.ccp:82/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&filter[channel_type_code][in][19]=vtt_cashtoutiao&filter[channel_type_code][in][20]=vtt_dianping&filter[channel_type_code][in][21]=vtt_weibo&filter[channel_type_code][in][22]=vtt_sina&filter[channel_type_code][in][23]=vtt_iqiyi&filter[channel_type_code][in][24]=vtt_toutiao_pro&filter[channel_type_code][in][25]=vtt_bilibili&filter[channel_type_code][in][26]=vtt_netease&filter[channel_type_code][in][27]=vtt_sogou&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753" |
6、将 [ 与 ] 前面添加 \,最终请求,响应成功。如图5
7、一直减少至 18 个参数后,就能够正常响应 200。总计为 918 个字符数。虽然官方 RFC 2616 没有指定限制,但许多安全协议和建议都规定服务器上的 maxQueryStrings 应设置为最大字符数限制 1024。而整个 URL,包括查询字符串,应设置为最大 2048 。
1 | {{host_info}}{{base_url}}/v{{module}}/task-group-channel-app-sources?filter[group_id]={{group_id}}&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&per-page=100&group_id={{group_id}} |
8、查看 Kong 的 tcp log。upstream sent too big header while reading response header from upstream。如图6
1 2 3 | 2021/09/30 09:10:12 [error] 65#0: *8025344 upstream sent too big header while reading response header from upstream, client: 10.42.0.231, server: kong, request: "GET /ccpapi/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753 HTTP/1.1", upstream: "http://10.103.175.193:82/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][ 15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753", host: "api.webtest.chinamcloud.cn" 10.42.0.231 - - [30/Sep/2021:09:10:12 +0000] "GET /ccpapi/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753 HTTP/1.1" 502 75 "-" "PostmanRuntime/7.28.4" |
9、在 kong 容器中添加环境变量。KONG_NGINX_PROXY_PROXY_BUFFER_SIZE、KONG_NGINX_PROXY_PROXY_BUFFERS。如图7
1 2 | KONG_NGINX_PROXY_PROXY_BUFFER_SIZE=160k KONG_NGINX_PROXY_PROXY_BUFFERS=64 160k |
10、再次请求,仍然响应 502,只不过响应体发生了变化,响应 HTML 。如图8
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | < html > < head > < title >502 Bad Gateway</ title > </ head > < body > < center > < h1 >502 Bad Gateway</ h1 > </ center > < hr > < center >nginx/1.17.10</ center > </ body > </ html > |
11、查看 Kong 的 tcp log。未再报错。如图9
1 | 10.42.0.231 - - [30/Sep/2021:10:34:36 +0000] "GET /ccpapi/v1/task-group-channel-app-sources?filter[group_id]=6c8b55e7387da4799772ddb389f06753&filter[channel_type_code][in][0]=vtt_douyin&filter[channel_type_code][in][1]=vtt_douyin_pro&filter[channel_type_code][in][2]=vtt_kuaishou&filter[channel_type_code][in][3]=vtt_kuaishou_pro&filter[channel_type_code][in][4]=vtt_baijia&filter[channel_type_code][in][5]=vtt_baijia_pro&filter[channel_type_code][in][6]=vtt_dayu&filter[channel_type_code][in][7]=vtt_dayu_pro&filter[channel_type_code][in][8]=vtt_qq&filter[channel_type_code][in][9]=vtt_yidian&filter[channel_type_code][in][10]=vtt_qutoutiao&filter[channel_type_code][in][11]=vtt_miaopai&filter[channel_type_code][in][12]=vtt_meipai&filter[channel_type_code][in][13]=vtt_rrtv&filter[channel_type_code][in][14]=vtt_sohutv&filter[channel_type_code][in][15]=vtt_sohu&filter[channel_type_code][in][16]=vtt_acfun&filter[channel_type_code][in][17]=vtt_yinyuetai&filter[channel_type_code][in][18]=vtt_neteasemusic&per-page=100&group_id=6c8b55e7387da4799772ddb389f06753 HTTP/1.1" 200 0 "-" "PostmanRuntime/7.28.4" |
12、参考网址:https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#proxy-buffers-number 。在 Rancher 中,查找到 kong 容器所对应的负载均衡,编辑 YAML 文件。添加 2 行 nginx.ingress 相关配置项。如图10
1 2 | nginx.ingress.kubernetes.io/proxy-buffer-size: 160k nginx.ingress.kubernetes.io/proxy-buffers-number: "64" |
13、再次请求,响应 200,符合预期。如图11
近期评论