å¦ä½ç¨ç¡¬ä»¶å éææ¯ä¸ºCPUåè´
ä¸ãèæ¯ä»ç»
éç¨å¤çå¨ï¼CPUï¼çæ©å°å®å¾å·²å ¥æ®å¹´ï¼èæºå¨å¦ä¹ åWebæå¡çè§æ¨¡å´å¨ææ°çº§å¢é¿ãä¼´éçå½ä»ç¡¬ä»¶ææ¯çæçåå±ï¼æ®éCPUæ 论æ¯å¨è®¡ç®è½åï¼è¿æ¯èµæºææ¬ä¸ç¸å¯¹äºä¸äºä¸ç¨ç¡¬ä»¶å·²ç»æ²¡æç»å¯¹ä¼å¿ï¼è¿ä¹ä¿ä½¿ç¡¬ä»¶å éææ¯å¾å°åå¤§å ¬å¸çéçï¼è¬å¦ä¸å¤§äºèç½å·¨å¤´ç¾åº¦ãé¿éãè ¾è®¯å é¨çæ¥å ¥å±éç¨ç±»ä¼¼KeyLessæ¹æ¡æ¥å éHTTPSçå¸è½½ï¼ä¸ä» æé«äºç¨æ·ä½éªï¼è¿èçäºæºå¨ææ¬ãæ ¹æ®å½åè°ç ç»æåç°ï¼ç®åä¸å åå¤§å ¬å¸æ¥å ¥å±é对äºGzipéç¨ç¡¬ä»¶å éè¿æ¯ä¸ç空ç½ï¼é¿éæ¥å ¥å±é¦æ¬¡ç»å硬件å éææ¯å¸è½½Gzipä¸ä» 带æ¥äºæ§è½æåï¼èä¸å¯¹ä¸çå¨æ¤é¢åçåå±ä¹æé大影åæä¹ã
æ¥å ¥å±Tengineå½åæ§è½ç¶é¢æ¯CPUï¼è¬å¦Gzip模åå¨Tengineä¸CPUå æ¯é«è¾¾15%-20%å·¦å³ï¼ç¸æ¯äºå ¶å®æ¨¡åCPUæ¶èé«ãå æ¯åå¢é¿è¶å¿ï¼å端åºç¨å缩é»è¾åç»ç»ä¸åç½®æ¥å ¥å±ï¼ãä¸éä¸ï¼æ以Gzip模å使ç¨ç¡¬ä»¶å¸è½½å¯¹äºæ§è½æåãææ¬ä¼åæ¯ä¸å¯æ缺ã
äºãåæä¸è°ç
åæåå ç®åä»ç»ä¸ä»ä¹æ¯ç¡¬ä»¶å éï¼ ç¡¬ä»¶å éï¼HardwareAccelerationï¼å°±æ¯å©ç¨ç¡¬ä»¶æ¨¡åæ¥æ¿ä»£è½¯ä»¶ç®æ³ä»¥å åå©ç¨ç¡¬ä»¶æåºæçå¿«éç¹æ§ï¼ç¡¬ä»¶å éé常æ¯è½¯ä»¶ç®æ³çæçè¦é«ï¼ï¼ä»èè¾¾å°æ§è½æåãææ¬ä¼åç®çï¼å½å主è¦æ¯å¦ä¸ä¸¤å¤§å éæ¹å¼ï¼
FPGA ç°åºå¯ç¼ç¨é¨éµåï¼å¯é对æä¸ªå ·ä½ç软件ç®æ³è¿è¡å®å¶åç¼ç¨ï¼è¬å¦ä¸å çæºè½ç½å¡ï¼
ASIC ä¸ç¨éæçµè·¯ï¼å®æ¯é¢åä¸é¨ç¨éççµè·¯ãä¸é¨ä¸ºä¸ä¸ªç¨æ·è®¾è®¡åå¶é çï¼è¬å¦IntelçQATå¡ä» æ¯æç¹å®å åå¯ãå缩ç®æ³ï¼
FPGAä¸ASICç对æ¯å¦ä¸è¡¨æ ¼æ示ï¼
2.1ãæ¥å ¥å±Tengine CPUæ¶èåæ
主ç«æ¥å ¥å±æ¿è½½éå¢90%以ä¸çå ¥å£æµéï¼çä¼¼åªæ¯ä½ä¸ºä¸ä¸ªä¸å±æµé转åç½å ³ï¼ä½æ¯å´åäºé常ä¹å¤çäºæ ï¼è¬å¦httpså¸è½½åå éãåå åãæºè½æµé转åçç¥ãç°åº¦åæµãéæµãå®å ¨é²æ»å»ãæµééåãé¾è·¯è¿½è¸ªã页é¢æç¹ççï¼è¿ä¸ç³»ååè½çèåæ¯Tengineä¼å¤æ¨¡åçæ¯æãç±äºåè½ç¹æ¯è¾å¤ï¼æ以è¿å°±å¯¼è´TengineçCPUæ¶èæ¯è¾åæ£ï¼å ¶ä¸»æµç¨å¤çå¦ä¸å¾æ示ï¼
å模åCPUæ¶èå æ¯Top 5å¦ä¸è¡¨æ ¼æ示ï¼
å°±å½åæ¥å ¥å±æµé模ååææ¥çï¼Gzipå个模åCPUæ¶èå æ¯è¾¾å°15%-20%å·¦å³ï¼æ³¨ï¼ä¸»è¦æ¯å缩æ¶èï¼ä¸å æ¯åä¸åè¶å¿ï¼æ以对Gzip使ç¨ç¡¬ä»¶å¸è½½è¿«å¨çç«ã
2.2ãå éæ¹æ¡è°ç
2.2.1ãIntel QATå¡
QAT(Quick Assist Technology)æ¯Intelå ¬å¸æ¨åºçä¸ç§ä¸ç¨ç¡¬ä»¶å éå¡ï¼ä¸ä» 对SSLé对称å 解å¯ç®æ³ï¼RSAãECDHãECDSAãDHãDSAçï¼å ·æå éï¼èä¸å¯¹æ°æ®çå缩ä¸è§£åä¹å ·æå éææï¼
QATå éå¡æä¾zlibå缩ç®æ³ãä¸zlib shimå¯¹å ¶åçzlibä¸QATä¹é´åäºéé ï¼è°ç¨æ¹å¼åzlibåºæ¹å¼åºæ¬ä¸è´ï¼éå¨ä¸å±ä¸å¡ä¸å¼å¯zlib QAT模å¼ãç¸å¯¹æ¥è¯´å¯¹ä¸å±ä¸å¡æ¹é è¾å°ã
2.2.2ãæºè½ç½å¡
INICï¼Intelligent Network Interface Cardï¼æ¯ç½ç»ç åäºä¸é¨èªç 产åï¼ä»¥ç½ç»å¤çå¨ä¸ºæ ¸å¿çé«æ§è½ç½ç»æ¥å ¥å¡ï¼å¯¹äºç½ç»æ¥ææ°æ®çå¤çé常åéï¼é对Tengineçgzipå¸è½½æå¦ä¸ä¸¤ç§æ¹æ¡ï¼
a. æä¾å缩APIç»hostï¼æå缩æ°æ®è¿åhostï¼ç±hostå°å åéï¼
b. hoståç½å¡çº¦å®å缩flagï¼hoståéæªå缩æ¥æï¼æºè½ç½å¡æ¶å°åè¿è¡å缩ï¼å¹¶ä¸éæ°å°å åéï¼
2.2.3ãFPGAå¡
FPGAï¼Field-Programmable Gate Arrayï¼ç°åºå¯ç¼ç¨é¨éµåï¼éè¦å¯¹æ¥å ¥å±ä½¿ç¨çzlibç®æ³ä½¿ç¨ç¡¬ä»¶è¯è¨éæ°å¼åãè¿è¡çµè·¯ç§åï¼ä¸ä¸å±äº¤äºé©±å¨ä¹éè¦ä»é¶å¼åï¼
æ¹æ¡å¯¹æ¯
æºè½ç½å¡çæ¹æ¡1ç¸æ¯äºQAT对zlibå¤ç没ææ§è½ä¸çä¼å¿ï¼æºè½ç½å¡åªæ¯å¯¹zlibè¿è¡è½¯ä»¶å¸è½½ãç¸å¯¹äºQAT并ä¸å ·æå éä½ç¨ï¼å ¶æ¹æ¡2éè¦æTengineä¸é¨åä¸å¡é»è¾æ½åå°ç½å¡ä¸åï¼å¦spdyãhttp2ãchunkedãssl对称å å¯ãååºbodyééçé»è¾ï¼å ¶ææ¬åé£é©é«ï¼æ¹æ¡3çFPGAæ¹å¼ç¸å¯¹æ¥è¯´å¼åææ¬è¾é«ãä¸ç¸å ³èµæºå®ä¹ã
综ä¸æè¿°æç»éç¨QATå éå¡å¯¹æ¥å ¥å±TengineçGzipè¿è¡å¸è½½ãå éã
ä¸ãæ¹æ¡å®æ½
QAT驱å¨éç¨UIOï¼UserspaceI/Oï¼ææ¯ï¼å ¶å¤§é¨åå¤äºç¨æ·æãåªæå°é¨åå¤ç硬件ä¸æåºççé»è¾å¤äºå æ ¸æï¼è¿æ ·ä¸ä» æ¹ä¾¿ç¨æ·è°è¯ï¼èä¸è¿è§£å³äºå æ ¸ä¸æ¯ææµ®ç¹æ°è¿ç®çé®é¢ãå½ç¶QATå éå¡ä¹é¡ºåºäºDockerèæåçæ½®æµï¼å ¶éç¨SRIOVææ¯ï¼å¯ä»¥å¨èææºä¹é´é«æå ±äº«PCIeï¼Peripheral Component Interconnect Expressï¼è®¾å¤ï¼å½åDH895XCCç³»åè¯çæé«å¯æ¯æ32个èææºå ±äº«QATï¼ä»èè¾¾å°å åå©ç¨ç¡¬ä»¶èµæºãå ¶æ¬¡QATå±äºASIC模å¼ç¸æ¯äºFPGAå ·ææ´å¥½çå éææï¼ä¸»è¦åå æ¯ç±äºFPGA为äºå¯éæï¼å¯¼è´å ¶é»è¾æ¥æ¾è¡¨ã触åå¨ä¼å¤ä»¥åç¸åé»è¾çµè·¯å¨å¸çº¿ä¸å»¶æ¶å大ã
æ¥å ¥å±Tengineç®åéç¨çæ¯ä¸å¾å·¦è¾¹çå®çº¿å éé¾è·¯ï¼å ¶ä¸Zlib ShimãQAT User Space ApiãQAT Driverä½ä¸ºTengine Gzipä¸åºå±ç¡¬ä»¶QATçéä¿¡éé å±ï¼æ¤æ¹å¼å¯¹ä¸å±ä¸å¡å ¥ä¾µè¾å°ãå ¶è½¯ä»¶æ¶æå¦ä¸å¾æ示ï¼
è½ç¶è¯¥æ¹æ¡çèµ·æ¥æ¯è¾ç®åï¼ä½æ¯çæ£çº¿ä¸å®æ½çæ¶åè¿æ¯éå°äºé常å¤çé®é¢ï¼åè½ãæ§è½æ¹é¢ï¼ï¼è¬å¦ï¼
3.1ãæ¶æä¸åç
a. 使ç¨ç第ä¸ç驱å¨Intel-Qat2.6.0-60ï¼å½QPS为1kå·¦å³æ¶CPUå¾å¿«æ满ï¼æ³¨ï¼æ£å¸¸æ åµä¸QPS为1kæ¶ï¼CPUæ¶è6%å·¦å³ï¼ï¼ä¸CPUæ¶èä¸90%以ä¸é½æ¯æ¶èå¨å æ ¸æï¼å¦ä¸å¾æ示ï¼
使ç¨straceè¿è¡ç¸å ³ç³»ç»çç¹å½æ°ç»è®¡åç°ï¼å ¶CPU主è¦æ¶èå¨ioctlç³»ç»å½æ°ä¸ï¼å¦ä¸æ示ï¼
éè¿perfæ¥çioctl主è¦æ¯æ§è¡å ååé å½ä»¤ï¼ç±äºZlib Shiméè¦å¼è¾è¿ç»çç©çå åãæ以åºç°é¢ç¹è°ç¨ compact_zoneè¿è¡å ç¢çæ´çï¼å ¶è°ç¨ççé«è¾¾88.096%ï¼å¦ä¸å¾æ示ï¼æ³¨ï¼ç度表示该å½æ°è¯¥å½æ°èªèº«çç度ãè°åºï¼ 表示被è°ç¨å½æ°çç度æ»åãæ»ä½ï¼ ç度 + è°åºï¼ï¼
åIntelç åèè°è®¨è®ºååç°æ¯ç±äºå½åIntel QATçZlib Shimç模åä¸åçæ导è´ï¼éè¿æ¨å¨å ¶æ¹é éç¨OOTçå å管ç模åUSDMï¼å é¨ç»´æ¤ä¸ä¸ªHugePageå åæ± ï¼æ¹æ¡è§£å³ã
b. 使ç¨ä¸è¿°é®é¢è§£å³åç驱å¨intel-qatOOT31092ï¼æµè¯ååç°CPUèçææä¸ä½³ï¼ç¨æ·æCPUåå°ãä½æ¯å¢å äºå æ ¸æçCPUï¼ï¼ç»åæãåç°ä½¿ç¨QATå éåï¼é¨åç³»ç»å½æ°CPUå æ¯åé«ï¼å¦ openãioctlãfutexï¼å¦ä¸å¾æ示ï¼æ³¨ï¼å·¦è¾¹çæ¯ä½¿ç¨QATååç³»ç»çç¹å½æ°ï¼ï¼ä½¿ç¨QATåopenãioctlãfutexæ§è¡æ¶é´å æ¯é«è¾¾8.95ï¼æ³¨ï¼3.91 + 2.68 + 2.36ï¼ï¼èæªä½¿ç¨çæ¬å¯¹åºå æ¯æ¶é´æ0.44ï¼æ³¨ï¼0.24 + 0.14 + 0.06ï¼ï¼
åæå ¶Tengineçworkerè¿ç¨å æ ä¿¡æ¯åç°openãioctlé½æ¯æ对åºç°ï¼å³ä¸æ¬¡http请æ±åºç°4次该系ç»è°ç¨ï¼ï¼è¯¥ç°è±¡åé¦ç»Intelçç ååå¦åå¾ç¥æ¯ç±äºæ°é©±å¨çZlib Shim导è´ï¼éè¿ä¼åæ¹é åopenãioctlè°ç¨é¢çææ¾åå°ãä½æ¯å ¶futexç³»ç»è°ç¨é¢åº¦å´æ²¡æåå°ï¼è¿æ¯å¯¼è´å æ ¸æçCPUå æ¯è¾é«ï¼éè¿straceè·è¸ªåç°ä¸ä¸ªhttpå缩请æ±åä¼å¤æ¬¡è°ç¨futexãå¦ä¸å¾æ示ï¼åIntelç ååå¦äºè§£å°Zlib Shiméç¨å¤çº¿ç¨æ¹å¼ï¼å ¶futexæä½æ¥èªzlib shimçå¾ QATå缩æ解å缩æ°æ®è¿åçé»è¾ã
ç±äºTengineæ¯å¤è¿ç¨å线ç¨ãéç¨epollå¼æ¥IOäºä»¶æ¨¡å¼ï¼èè°Intelçç ååå¦å¯¹Zlib Shimè¿è¡æ¹é ï¼å»çº¿ç¨ï¼ï¼æç»futexç³»ç»è°ç¨ä¹ææ¾åå°ã
éè¿åæ并æ¨å¨Intel对QATè¿è¡å¤æ¬¡æ¶æä¸çæ¹é ï¼æ使å¾QATçå éç¹æ§æ´å¥½çåæ¥ã
3.2ãåè½ä¸å®å
a. 使ç¨QATåæ§è¡reloadï¼å¯è½å¯¼è´è¯·æ±ååºå¼å¸¸ï¼å¦ä¸æ示ï¼
ç±äºæ¯ä¸ªworkerè¿ç¨é½éè¦åé ä¸ä¸ªQAT Instanceç¨äºæ°æ®è§£å缩ï¼Tengineå¨reloadçç¬é´workerè¿ç¨æ°å¯è½ä¼ç¿»åãèQAT Instanceåå§çæ¬åªæ64个ãæ以æ°å¯å¨çworkerè¿ç¨å¯è½åé ä¸å°Instanceã导è´è¯·æ±å¤±è´¥ã
é对æ¤é®é¢Intelæä¾çæ°çæ¬QATï¼å ¶Instanceæ°éä»64æé«å°256个é¿å æ¤é®é¢çåçï¼åæ¶æ们æåºå®¹ç¾ä¿æ¤æ¹æ¡ï¼å½Instanceæ æ³åé äºéè¦èªå¨é级为软件å缩ï¼æé«å ¶å¯ç¨æ§ã
b. Zlib Shim huge pageå åæ³æ¼ï¼å¯¼è´QAT驱å¨core dumpï¼
Tengine使ç¨å åæ± æ¨¡å¼è¿è¡å åç管çï¼å³è°ç¨(In)DeflateInitåé ç空é´æ éè°ç¨(In)DeflateEndå¤çãå¨è¯·æ±ç»æçæ¶åä¼è°ç¨è¯·æ±rç¸å ³çéæ¾æä½ï¼è¿è¡å åçå½è¿ï¼ä½æ¯ç±äºZlib Shim使ç¨çhuge pageå¿ é¡»è°ç¨(In)DeflateEndæéæ¾ç»USDMï¼éè¿æ¹é Tengine Gzipç¸å ³ä»£ç åï¼è¯¥é®é¢å¾ä»¥è§£å³ï¼èQAT驱å¨çcore dumpä¹æ¯ç±äºhugepageçæ³æ¼å¯¼è´æ æ³æååé 导è´ã
c. Zlib Shimç¶ææºä¸å®å导è´ç¹å®åºæ¯ä¸çå缩ã解å缩请æ±å¼å¸¸ï¼çä¼å¤é®é¢å°±ä¸ä¸ä¸ä»ç»ã
ä¸è·¯èµ°æ¥ï¼éè¿æ æ°æ¬¡çæ§è½ä¼åãåè½æµè¯ï¼å¤æ¬¡åIntelç ååå¦ä¸èµ·æ¢è®¨ä¹åï¼æ使å¾QATå¨åè½ãæ§è½ãæ¶ææ¹é¢çä¼å¤é®é¢å¾ä»¥å¿«é解å³ï¼ä¸é¢å°±åå¤ä¸çº¿åæåå¤å·¥ä½ã
3.3ãè¿ç»´æ¢³ç
é¨ç½²åå¸
éç¨årpm软件å ãåäºè¿å¶æ¨¡å¼ï¼ä»èéä½è½¯ä»¶çä¸ç¡¬ä»¶å éçä¹é´çè¦å度ï¼èªå¨è¯å«é¨ç½²æºå¨æ¯å¦å¼å¯QATï¼å¹¶éæ©æ£ç¡®çäºè¿å¶æ§è¡ï¼
容ç¾ä¿æ¤
è¿è¡è¿ç¨ä¸ç±äºæç§èµæºç缺ä¹å¯¼è´ç¡¬ä»¶å éçæ¬Gzipæ§è¡å¤±è´¥ï¼å°ä¼èªå¨åæ¢ä¸ºè½¯ä»¶çæ¬ãå¾ èµæºå¯ç¨æ¶èªå¨åæ¢å°ç¡¬ä»¶å éçæ¬ï¼
å¯ç»´æ¤ä¸çæ§
è½ç¶ä¸çº¿ååè¿ä¸ç³»ååæµã稳å®æ§å¹¶æªåºç°å¼å¸¸ï¼ä½å¯¹ç¡¬ä»¶å éçç¸å ³èµæºææ è¿è¡å®æ¶çæ§è¿æ¯å¿ ä¸å¯å°ï¼
åãå éææ
æµè¯æºå¨
cpuåå·ï¼Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz 32æ ¸ å æ ¸ï¼2.6.32 Zlibçæ¬ï¼zlib-1.2.8 QAT驱å¨çæ¬ï¼intel-qatOOT40052
æ°æ®å¯¹æ¯
åçæ¡ä»¶ä¸ï¼å¼å¯QATå éåCPUå¹³åå¼ä¸º41%å·¦å³ï¼æªå¼å¯QATå éçCPUå¹³åå¼ä¸º48%å·¦å³ï¼å¦ä¸å¾æ示ï¼
ç¸åæ¡ä»¶ä¸ï¼å¼å¯QATå éåç³»ç»loadå¹³åå¼ä¸º12.09ï¼å ³éQATå éæ¶ç³»ç»loadå¹³åå¼ä¸º14.22ï¼å¦ä¸å¾æ示ï¼
ç¸åæ¡ä»¶ä¸ï¼å¼å¯ä¸å ³éQATå éåï¼ååºRTæ³¢å¨ä¸ç¸ä¸ä¸ï¼å¦ä¸æ示ï¼
åçæ¡ä»¶ä¸ï¼å模åçç¹å½æ°å¾å¯¹æ¯å¦ä¸æ示ï¼å ¶ä¸çº¢è²åä¸çæ¯Gzipç¸å ³å½æ°
ï¼æ³¨ï¼å·¦ä¾§æ¯å¼å¯QATå éï¼ï¼
åæ¯æ¡ä»¶ä¸Tengine Gzip使ç¨QATå éå¡åï¼CPUæ¶èä»48%ä¸éå°41%ï¼ç³»ç»è´è½½loadä¸é2个ï¼ä¸æ ¹æ®æ¨¡åçç¹å½æ°å¾å¯¹æ¯åç°Gzipåºæ¬ä¸å·²ç»å®å ¨å¸è½½ã
ç»è®º
综ä¸æ°æ®å¯¹æ¯ï¼å½qps为10kå·¦å³æ¶Tengine Gzip使ç¨QATå éåCPUèç15%å·¦å³ï¼ä¸Gzipåºæ¬ä¸å®å ¨å¸è½½ãéçå ¶å æ¯åé«ï¼ä¼åææå°è¶å¥½ã
äºãæ»ç»
æ¥å ¥å±Tengine Gzip硬件å é项ç®æ¯é¿éåå¨ææ¯Tair&Tengineå¢éåæå¡å¨ç å计ç®å¢éä¸è±ç¹å°æ°æ®ä¸å¿ç½ç»å¹³å°å¢éé½å¿ååä¸ç产ç©ï¼ä¸ä» 带æ¥äºæ§è½æåï¼èä¸ä½¿å¾æ¥å ¥å±å¨ç¡¬ä»¶å éé¢åå次æä¸äºåå®çåºç¡ã为æå¹´SSL+Gzipæ¶æä¸æ´åå好äºæ²æ·ï¼åæ¶ä¹å¡«å äºä¸å æ¥å ¥å±å¯¹Gzipéç¨ç¡¬ä»¶å éç空ç½ï¼å¯¹æ¤é¢åçåå±å ·æä¸å®çå½±åæä¹