ishenwei/nexus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Auto-sync: 2026-04-21 17:12

Browse Source
This commit is contained in:
weishen
2026-04-21 17:12:45 +08:00
parent 914c8f6925
commit 0fe7ba237f
1888 changed files with 220 additions and 68174 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
wiki/concepts/Token.md
View File

@@ -1,18 +0,0 @@
---
title: "Token"
type: concept
tags: [llm, token]
date: 2025-12-20
---
## Definition
大模型各种算法的基本输入单元，可以认为是一个单词或者一个短语。
## Token Calculation Rules
- 1 个英文字符 ≈ 0.3 个 token
- 1 个中文字符 ≈ 0.6 个 token
## Related Concepts
- [[LLM]]：使用 Token 作为输入的语言模型
- [[Embedding]]：将 Token 转化为向量化表示
- [[vLLM]]：优化 Token 处理效率的推理引擎